Introduce '$self' macro metavar for hygienic macro items

[SergioBenitez]

This RFC introduces the $self macro metavariable, a companion to $crate, that allows macros hygienic access to items. With $self, a macro m could be declared as:

mod submod {
    static PRIVATE: &'static str = "PRIVATE_SUBMOD";

    #[macro_export]
    macro_rules! m {
        () => (println!("{}", $self::PRIVATE))
    }

    pub use m;
}

pub fn main() {
    submod::m!(); // `PRIVATE` unconditionally resolves to `submod::PRIVATE`
}

On expansion of m, PRIVATE unconditionally resolves as if it were at the definition site, that is, to submod::PRIVATE.

Rendered

[tesuji]

At first glance, I was thinking $self help us write <item>.macro!() (the suffix postfix macro).

[SergioBenitez]

At first glance, I was thinking $self help us write <item>.macro!() (the suffix macro).

I don't think this misunderstanding is a concern: $self doesn't have that meaning today, and declarative macros are always function-like (f!(), f![], f!{}), which this RFC doesn't modify. Furthermore, this RFC does not modify what it means for $self to appear as a user-declared metavariable; it only modifies the meaning of code that was previously invalid.

[lebensterben]

This RFC introduces the $self macro metavariable, a companion to $crate, that allows macros hygienic access to items. With $self, a macro m could be declared as: ...

@SergioBenitez Rustc dev guide book says:
When expanding macros, name resolution is involved for imports and macro names. But other names are NOT resolved yet.
See https://rustc-dev-guide.rust-lang.org/macro-expansion.html#name-resolution

To make the proposed RFC work, it seems to me that it will break the current order of name resolution.

[ogoffart]

Postfix macros do not exist yet, but it has been proposed in #2442 and it propose to use $self as well:

you can include a first argument that uses a designator of self (typically $self:self).

While not completely incompatible, they still propose a conflicting purpose for the idiomatic use of $self in macro. Since both RFC are desirable features (IMHO) it would be nice if they could play well together.

$self comes as an natural keyword for referring to the module. But if we had to find an alternative, I could find:

• $mod : which is analogue to $crate but unfortunately is not what is used in normal paths. (Although one could imagine extending the language so mod::foo::bar would mean the same as self::foo::bar)
• $super : but would that be the current module, or the parent module as it is in normal paths?

---

Another thing, while this rfc state that

unlike $crate, $self does have an effect on visibility

would it not make sense to "update" the meaning of $crate to be able to refer to pub(crate) items from within the macro?
This is an orthogonal change, bout would make the use of $crate and $self more consistent, i think.

[matklad]

One thing I've realized that this (as well as macro 2.0) has an interesting implication for IDE support.

One of the core data structures of an IDE is map which maps each crate to a set of items this crate contains. In rust-analyzer, this is the heaviest data structure.

One obvious optimization would be to include only publicly-visible API in this map for dependencies. This would cut down the size significantly directly. Moreover, it should help with incremental compilation --- you know that, if the public interface of the crate hasn't changed, you don't have to re-compile it's reverse dependencies.

Note that neither of the two optimizations are implemented yet, but they seems both plausible and relatively important in terms of impact. From this point of view, the current setup where you have to mark items used by macros across crates beneficial.

A plausible solution here is to require marking of macro-accessible cross-crate public items:

mod submod {
    pub(macro) static PRIVATE: &'static str = "PRIVATE_SUBMOD";

    #[macro_export]
    macro_rules! m {
        () => (println!("{}", $self::PRIVATE))
    }

    pub use m;
}

Note that there's no problems withing a single crate, as there you need unabridged map anyhow.

you know that, if the public interface of the crate hasn't changed, you don't have to re-compile it's reverse dependencies.

To clarify, this is in terms of cargo check compilation, for full compilation with inlining and such, you obviously need to know private impl details.

[Aaron1011]

Can a $self keyword only be generated by macro_rules! macro (i.e. it cannot be produced by proc-macros)?

[Aaron1011]

Instead of Span::source(), I think it would make more sense to talk about Span::def_site(). That is, using a $self path (conceptually) first substitutes in the path to the macro's parent module, and then behaves as if a proc-macro had produced the tokens with Span::resolved_at(Span::def_site())

[SergioBenitez]

Unfortunately, I don't think Span::def_site() is quite right either. For example, in the expansion chain #[attr] -> macro_rules foo!() { $self }, $self should resolve wherever #[attr] was called (or equivalently, where the tokens macro_rules foo!() { $self } expanded to), not where #[attr] was defined.

I believe the precise way to phrase this is:

Thus, $self can simply and without further caveats by specified as: for
every path in the expansion that begins with $self, the resolution context of
the path is set to resolution context of the first macro_rules! definition in which
the $self tokens appeared.

Then proceed to show, as examples, 1) nested macro_rules!, where the outer scope is used, and 2) a proc-macro that expands to a macro_rules!, where the call-site of the proc-macro is used.

[Aaron1011]

Sorry for being unclear - I meant that the $self path should be resolved as if it was produced by a proc-macro with Span::def_site(), instead of a macro_rules! macro (after the parent module path has been substituted).

[Aaron1011]

What should the behavior of the following code be?

struct Bar;
macro_rules! outer {
    () => {
        #[macro_export]
        macro_rules! inner {
            () => { struct Foo($self::Bar); }
        }
    }
}

mod other_mod {
    outer!();
    inner!();
}

For consistency with $crate, I think this should compile - that is, $self will end up referring to the 'outermost' macro_rules! definition. This is fairly subtle, so I think it's worth explicitly mentioning in the RFC.

[SergioBenitez]

@ogoffart

While not completely incompatible, they still propose a conflicting purpose for the idiomatic use of $self in macro. Since both RFC are desirable features (IMHO) it would be nice if they could play well together.

I don't think the two proposals conflict at all! $self as $self:self could be named anything, including $self! If it is named $self, then it unambiguously refers to the user-declared metavar. Otherwise, the appearance of $self works as described in this RFC. This is identical to the way $crate works today. Of course, should both RFCs be accepted, documentation should likely see an alternative to $self be used as the canonical name for self metavariable types. Perhaps $this or $e would be good candidates.

would it not make sense to "update" the meaning of $crate to be able to refer to pub(crate) items from within the macro? This is an orthogonal change, bout would make the use of $crate and $self more consistent, i think.

I like the idea. I think we should keep this particular RFC focused on $self, however. I'd be happy to draft a follow-up!

@matklad

A plausible solution here is to require marking of macro-accessible cross-crate public items: [...] pub(macro).

Besides semantic issues with pub(macro) (can macros outside the crate also access pub(macro) items?), the need to mark items with pub(macro) obviates one of the primary purposes of this PR: to allow macros to "capture" their scope. In particular, one of the main reasons (see the example later in this comment) for this RFC is to allow proc-macros to generate macro_rules! macros that can refer to local, private items by direct item name without any additional effort from the user. Furthermore, requiring pub(macro) would also require explicitly importing prelude types to mark with pub(macro), which this RFC does not.

@Aaron1011

For consistency with $crate, I think this should compile - that is, $self will end up referring to the 'outermost' macro_rules! definition. This is fairly subtle, so I think it's worth explicitly mentioning in the RFC.

Agreed. This is exactly the intention: $self refers to the scope of the first macro_rules! in the expansion chain in which it appeared.

Can a $self keyword only be generated by macro_rules! macro (i.e. it cannot be produced by proc-macros)?

It can be produced by proc-macros that generate macro_rules! macros. Combined with the definition above, a chain like #[attr] -> macro_rules! foo(), where foo includes $self, would see $self resolve as if at the call site of #[attr]. As an example, while your code example would resolve, the following would not:

struct Bar;

#[proc_macro]
fn outer(_: TokenStream) -> TokenStream {
    quote! {
        #[macro_export]
        macro_rules! inner {
            () => { struct Foo($self::Bar); }
        }
    }
}

mod other_mod {
    outer!();
    inner!();
}

But the following would:

#[proc_macro]
fn outer(_: TokenStream) -> TokenStream {
    quote! {
        #[macro_export]
        macro_rules! inner {
            () => { struct Foo($self::Bar); }
        }
    }
}

mod other_mod {
    struct Bar;
    outer!();
    inner!();
}

@Aaron1011 If this make sense, I can update the RFC text accordingly.

[Aaron1011]

@Aaron1011 If this make sense, I can update the RFC text accordingly.

That makes sense to me - the quote! examples would produce $ self, which would then be parsed to a single $self ident when the macro_rules! macro gets defined.

[withoutboats]

We discussed this briefly in triage this week.

We are uncertain about allowing access to private members visible to def-site right now; it feels like a bigger addition to macro capabilities than we'd want to make under the 2021 roadmap.

However, we are more comfortable with a more limited form of $self, which expands to $crate::<path-to-self>, and causes an error if the macro call site can't access the full path, just like a path prefixed with $crate would.

We also discussed whether this behavior would also make self to apply to the $crate path as well. There's a logical connection between $self and def-site visibility, but we didn't see an inherent reason $crate couldn't have the same visibility support.

[SergioBenitez]

@withoutboats

We are uncertain about allowing access to private members visible to def-site right now; it feels like a bigger addition to macro capabilities than we'd want to make under the 2021 roadmap.

However, we are more comfortable with a more limited form of $self, which expands to $crate::<path-to-self>, and causes an error if the macro call site can't access the full path, just like a path prefixed with $crate would.

Can you tell me more about where uncertainty is arising? Access to def-site hygiene is my primary objective with this RFC, and so I'd be rather disappointed to see it go by the wayside. The RFC as proposed is fully backwards compatible, so waiting for 2021 seems unnecessary. What's more, this RFC merely pushes forward already accepted ideas from decl_macro, ideas that have been discussed at length by the community, have successfully gone through several RFC processes, and which, if not for missing implementation work, would likely otherwise be stabilized. As such, given it is fully backwards-compatible, and given that its ideas have already been vetted and approved, I'd consider this RFC to be as risk-free as it gets.

To be concrete, there are at least two major usability issues in Rocket that this RFC would resolve. I'll explain the most prominent one now, rwf2/Rocket#1120. In short, to support Rust stable, where we used to generate a decl_macro, we now generate a macro_rules! macro at the call-site of an #[attr] macro. The macro allows for generating type-safe URIs based on the types in the user's route function, decorated with #[attr]. That macro can be called from anywhere in the Rocket application, including outside of the module where it was defined as result of the expansion of the #[attr] macro. The macro refers to types in the route function. Without this RFC, those types need to be in-scope at the call-site, which is remarkably counter-intuitive. With this RFC, the generated macro would be able to capture the types at the def-site.

We also discussed whether this behavior would also make self to apply to the $crate path as well. There's a logical connection between $self and def-site visibility, but we didn't see an inherent reason $crate couldn't have the same visibility support.

I agree, and I'd be happy to amend the RFC to give $crate paths visibility to crate-level items if it helps the RFC process.

[Aaron1011]

I agree, and I'd be happy to amend the RFC to give $crate paths visibility to crate-level items if it helps the RFC process.

Would that become the default behavior of $crate paths? If so, that seems like a potential Semver hazard. If a crate exposes a macro that does something like $crate::$name with a caller-provided $name, downstream crates will be able to access private items.

[matklad]

What's more, this RFC merely pushes forward already accepted ideas from decl_macro, ideas that have been discussed at length by the community, have successfully gone through several RFC processes, and which, if not for missing implementation work, would likely otherwise be stabilized.

@SergioBenitez I didn't find the mentioned discussions/RFCs after some moderately thorough googling. Could you provide the links for people like me who haven't been following the development closely?

I've found #1584 ("placeholder" RFC for declarative macros) and https://github.com/jseyfried/rfcs/blob/hygiene/text/0000-hygiene.md (hygiene RFC text which, to my knowledge, was never submitted as an RFC, and didn't go through the RFC process).

[petrochenkov]

@matklad

One obvious optimization would be to include only publicly-visible API in this map for dependencies. This would cut down the size significantly directly.

We have issues with optimizations like this in macros 2.0, which allow cross-crate access to private items.
During codegen we have to mark entities as "exported" so they are available at link time, and such marking is currently very pessimistic if any public macro 2.0 macros are in scope, any private entities accessible from that macro's location have to be exported.

[petrochenkov]

Assuming $self is identical to self module with def-site hygiene, similarly to $crate and def-site crate (just to avoid introducing additional subtly different kinds of hygiene, which the RFC as written does):

• The privacy component of this proposal requires a more significant part of macro 2.0 machinery to be available and stable in the compiler.
• The resolution component ($self == self resolved at def site) requires a less significant part of macro 2.0 machinery. Until very recently, the main missing part was cross-crate hygiene, but the first iteration of it was recently implemented. Not sure which other parts of macros 2.0 are prerequisites for this, need to think a bit.

[petrochenkov]

There's a tiny bit of backward incompatibility here that affects any possible new $keywords.
$keyword turns from two tokens to one:

macro_rules! takes_two_tokens {
    ($dollar:tt $self:tt) => {}
}

macro_rules! check {
    () => {
        takes_two_tokens!{$self} // Currently OK, but will break
        takes_two_tokens!{$crate} // Currently not OK, "processed" $crate is a single identifier token
    }
}

check!();

fn main() {}

[pksunkara]

Would you say that if this RFC is approved and implemented, it will be released with the 2021 edition because of that backward incompatibility?

Also, I would vote for $mod or $super

[durka]

Putting in my vote for $mod. It will be familiar by analogy to $crate. $self on the other hand is really confusing and sounds like macro-methods or something (and, IMO, should be saved in case we eventually do something along those lines).

[rpjohnst]

$crate:: matches crate::; $self:: matches self::. There is no mod::, and worse there is already a super:: that means something else.

[nikomatsakis]

We discussed this in today's backlog bonanza, but our conclusion was roughly the same as what @withoutboats already wrote some time back:

We are uncertain about allowing access to private members visible to def-site right now; it feels like a bigger addition to macro capabilities than we'd want to make under the 2021 roadmap.

In short, tackling and exposing hygiene feels like a bigger project and one that we don't currently have enough expertise to tackle (and this is basically a hygiene system). Right now @petrochenkov definitely understands that system best, the main problem is that we haven't made much progress in spreading understanding of the current system or documenting how it works. As @petrochenkov noted:

The privacy component of this proposal requires a more significant part of macro 2.0 machinery to be available and stable in the compiler.

and this feels like a decent chunk of work to vet and manage that we'd have to schedule carefully.

There remains some interest in $self (and presumably $super) as nice sugar, though in our conversation we hadn't fully read through the rfc thread yet and so the backwards compatibility issues that @petrochenkov raised were not discussed.

[joshtriplett]

Following up, we'd love to see this RFC partitioned into two chunks:

• A version of this RFC that just introduces $self, and $super, without affecting hygiene. This just needs a simple RFC.
• A project group proposal, to sort out macros 2.0 in general, and bring it over the finish line. This needs a lang MCP.

[joshtriplett]

In addition to the backwards compatibility issues that @petrochenkov raised, there's an additional compatibility issue that I didn't see discussed in this thread: today, you cannot write a macro parameter named $crate, but you can write one named $self (or $super). For instance:

macro_rules! mac {
    ($super:expr) => { $super + $super }
}

fn main() {
    let x = mac!(2 * 3);
    println!("{}", x);
}

If we gave $super or $self a specific meaning, this would potentially introduce conflicts. In particular, it is possible for $super::some::path or $self::some::path to appear in a macro today, given a macro parameter that could form the start of a qualified path.

@SergioBenitez

If it is named $self, then it unambiguously refers to the user-declared metavar. Otherwise, the appearance of $self works as described in this RFC. This is identical to the way $crate works today.

As far as I can tell, $crate does not work this way today. For instance, the following does not compile:

macro_rules! mac {
    ($crate:expr) => { $crate + $crate }
}

fn main() {
    let x = mac!(2 * 3);
    println!("{}", x);
}

[joshtriplett]

Following up on this: it turns out that quite a few crates currently use $self as a macro parameter. We're hesitant to introduce that level of breakage, even in an edition.

One alternative we're looking at for extending macro syntax would be something like $$self and $$super (and we could introduce $$crate as well to have parallel syntax). Since that isn't currently valid syntax, it can't conflict with anything.

[programmerjake]

Maybe while we're at it, we could add a way of escaping $ to allow doing something like this:

macro_rules! m {
    ($v:ident) => {
        assert_eq!(
            stringify!($$v), // or some other escaping syntax, maybe `$dollar`
            "$v",
        );
    };
}

[mbartlett21]

Would $Self be better?
It would match better with the current use in an impl for the implemented type.