‘Designing institutions and coordination mechanisms’ is long but pretty good, I think? It somewhat clarifies for me what I think the intended focus is, in a way that ‘institution design’ on its own doesn’t. ‘Designing X’ is also less formal than ‘X design’, which suggests less that we’re trying to supersede the standard academic terminology.

If ‘mechanism design’ isn’t ideal, adding some qualifier might fix it, like ‘social mechanism design’ or ‘economic mechanism design’.

Or you could do something like ‘group and institution design’.

I suggest being mindful of both the denotations and connotations of the names chosen, since this may influence the direction of future discussion. E.g., ‘designing institutions’ naturally calls to mind for me ‘designing large organizations’, but it’s less evocative to me of ‘designing norms for LW meetup’ or ‘designing good political voting systems’ (even though both of these apparently can fall under ‘mechanism design’). Other terms, like ‘group design’ or ‘coordination mechanism design’, will make different things salient.