Does metals need to be re-architected?

[vishalovercome]

In the past, I've raised various issues faced while working on a mixed Scala-Java project and while the maintainers have been prompt in replying, more often than not my issues remain unresolved. Yesterday, this discussion was initiated by somebody else and I'm going to start a different kind of discussion because I can see a much bigger problem with current architecture as well as style of development.

In short, I think that the direction of Metals is at odds with the direction of Scala community. These are some of the projects that community is working on:

1. Scala projects built against various targets
   1. Scala-js
   2. Scalapy
   3. Scala Native
   4. JVM
   5. GraalVM
2. Rise in adoption of fiber based effect systems
   1. Zio
   2. Cats-effect
   3. Monix
3. Multiple build tools
   1. sbt
   2. bloop
   3. pants
   4. bazel
4. Multiple development environments
   1. LSP based (vscode, neovim)
   2. Non-LSP (IntelliJ, ensime (emacs)) - outside the purview of Metals, but is influenced by it
5. Scripting
   1. Scala-cli
   2. Ammonite
   3. Worksheets

There are plenty of other dimensions where community gets split (scala 2 vs scala 3, testing libraries, etc.). In the future performance profiling may become an integral part of an VSCode. So far, it's been the job of metals developers to address the entire community. But is that sustainable? Can metals truly keep up with all the projects that are being worked on? I can see the list of features piling up all the time.

When it came to Java support, I got the following responses:

1. Use IntelliJ as it has better mixed Java-Scala support
2. RedHat LSP and Metals LSP can't be made to work together
3. Metals can't offer full Java LSP support
4. Key debugging features (conditional breakpoints, logpoints, watch, expression evaluation) can't work. Some of these aren't even supported for Scala`

Going by this, question would be how will Metals support a very promising project such as scalapy? Won't the same set of questions come up again? Providing a first class developer experience for various build targets, specialized frameworks, build tools, etc. will never ever be possible because 1 team can never support the use cases for the plethora of tools.

For this reason, the core metals team should provide a long term roadmap that's time bound as well. It will then be apparent that the current architecture/style of development is too centralized and that Metals should have a plugin based architecture which allows for various other LSPs, framework specific tooling to be integrated into it by people other than the core metals team.

I don't know what such an architecture ought to be, in part because the architecture and code of Metals isn't really well documented. What I can see is that the team is working on features that ideally ought to be done by maintainers of other projects. Scala 3 support for instance should be done by the compiler team. Likewise, Scalapy/Zio/testing library/etc. support by the developers of the respective projects. Only then can Metals keep up with rest of the community.

To summarize, metals should provide:

1. Focus on better IDE specific features (first class debugging, refactoring, etc.)
2. Provide generic support for each category of features
3. Provide necessary documentation and interfaces that allow others to plugin their projects

Would love to know what others think about Metals support for the entire Scala ecosystem!

[tanishiking]

Thanks @vishalovercome for raising this!

I believe you point is

• Metals developers should focus on "core features"
  • by designing Metals to be pluggable architecture, and let other developers can build plugins.
• Metals project should provide long-term roadmap
• Metals should provide better documentation about architecture

am I correct?

---

It will then be apparent that the current architecture/style of development is too centralized and that Metals should have a plugin based architecture which allows for various other LSPs
framework specific tooling to be integrated into it by people other than the core metals team.

Yeah, I also dream about making Metals to be plugin-architecture, and let users can develop their own features using all the semantic information / build information / syntctic information gathered by Metals 😄
However, I have no idea how can we achieve that kind of architecture in IDEs (what kind of APIs and hook timing would be really valuable to users?). Any ideas are welcome! (maybe we can borrow some ideas from eclipse jdt? just a hunch)

For LSP features, yeah, that would be awesome if we can scalameta/metals-feature-requests#5

---

For this reason, the core metals team should provide a long term roadmap that's time bound as well.

Sorry, I couldn't catch why you came up to this conclusion. Could you elaborate on why you think providing a roadmap will help the problem you mentioned?

While providing a long-term roadmap would be ideal (currently, Metals actually don't have a roadmap 😆 ), but I don't think it's a good idea to provide a road map that's time-bound for OSS in general.
Even though some companies back this project, it's pretty hard to estimate how much development resources will be assigned to OSS projects. On top of that, OSS projects have a tons of interruption or small tasks such as small bugfixes, answering issues, and so on.

---

I don't know what such an architecture ought to be, in part because the architecture and code of Metals isn't really well documented.

Which part do you think is not well-documented?
I wrote up some documentation something called architecture.md so someone who's new to the project can grasp the architecture and feature locations.
We need feedback to improve the documentation, any feedbacks are welcome :)

---

What I can see is that the team is working on features that ideally ought to be done by maintainers of other projects. Scala 3 support for instance should be done by the compiler team. Likewise, Scalapy/Zio/testing library/etc. support by the developers of the respective projects.

I don't think so actually (except if we're going to re-implement Java language features from scratch 😅 ).

• Scala 2 and 3 compiler APIs are mostly developed by compiler teams. So we don't need to implement Scala type checker by ourselves.
  • I believe, for example, advanced completion features using compiler API, should be maintained by IDEs, it's not compiler's responsibility.
• Each build server support is done by sbt / mill / Bazel(-BSP) teams and communicates over BSP

What part do you think the Metals team should delegate to others, for example? (Maybe test explorer stuff?)

[vishalovercome]

Thanks @vishalovercome for raising this!

I believe you point is

• Metals developers should focus on "core features"

  • by designing Metals to be pluggable architecture, and let other developers can build plugins.

• Metals project should provide long-term roadmap

• Metals should provide better documentation about architecture

am I correct?

It will then be apparent that the current architecture/style of development is too centralized and that Metals should have a plugin based architecture which allows for various other LSPs
framework specific tooling to be integrated into it by people other than the core metals team.

Yeah, I also dream about making Metals to be plugin-architecture, and let users can develop their own features using all the semantic information / build information / syntctic information gathered by Metals 😄 However, I have no idea how can we achieve that kind of architecture in IDEs (what kind of APIs and hook timing would be really valuable to users?). Any ideas are welcome! (maybe we can borrow some ideas from eclipse jdt? just a hunch)

For LSP features, yeah, that would be awesome if we can scalameta/metals-feature-requests#5

Yes. This is exactly what I want!

[vishalovercome]

Right after asking for "time bound roadmap" I had mentioned that it would be apparent that metals team would never be able to keep up with the pace of the community.

My goal isn't to get timelines. Instead, I want the team to take some time away from development and rethink a strategy that allows for wide variety of support that will be demanded from metals. Just Java support took so long and is hardly useful. Now imagine Scalapy/ScalaJS/Scala Native! Then someday, we'll talk about frameworks and then libraries!

Better to halt all development and provide a strategy to move forward.

[dos65]

@vishalovercome I read this post several times but honestly I don't see the constructive basis on suggestions to take some actions that you suggested.

From full-time maintainers side I can say right now that the amount of people and time allows us to work mainly on Scala features.
The current roadmap is to complete Scala3 support to provide the same set of features that we have for Scala2 + do some set of fixes for Scala that will be turned into 1.0.0 release. That's the plan.

For this reason, the core metals team should provide a long term roadmap that's time bound as well. It will then be apparent that the current architecture/style of development is too centralized and that Metals should have a plugin based architecture which allows for various other LSPs, framework specific tooling to be integrated into it by people other than the core metals team.

There is no issue with centralization/architecture.
I might be wrong, but it feels like you think that if we introduce some plugin system it will magically result that people start bringing the features or fixes some long-standing issues.
Unfortunately, it's not the case. For example, the issue you mention about java support just a tough one and requires a lot of time.
There is no smart strategy that might turn this issue into an easy one. To see it being solved somebody have to find time and desire to implement it.

[ckipp01]

@vishalovercome I read this post several times but honestly I don't see the constructive basis on suggestions to take some actions that you suggested.

I just wanted to support this notion.

Apart from what @dos65 has already mentioned let me add a couple more thoughts.

So far, it's been the job of metals developers to address the entire community. But is that sustainable? Can metals truly keep up with all the projects that are being worked on? I can see the list of features piling up all the time.

In short, no I don't think Metals can, but I also don't think Metals should. While having better integration with tools like Scalapy would be great, keep in mind that the number of people that would benefit from an integration like this is incredibly small. I'm even on the far end of the spectrum that I don't think Metals should have a deeper Java/Scala integration. I have no problem recommending users to jump to IntelliJ for projects that are heavy mixes of Scala/Java/Kotlin. Imo that's not the aim of Metals, or at least it wasn't. I wouldn't never recommending stagnating Scala support or pure Scala features for a deeper integration with another language. At it's core, Metals is a Scala language server.

I do agree that there are improvements we could make that would make the experience better for cross building, cross-platform support, etc. These are very good goals, however I don't think they require a core architectural change. I think you may be underestimating the amount of work that would be.

I'd like to think that as a team, we're very open to feedback, community input, and doing everything we can to make the experience of a Metals user better. However, I must admit that statements like:

I can see a much bigger problem with current architecture as well as style of development

Just Java support took so long and is hardly useful.

Better to halt all development and provide a strategy to move forward.

Make me a bit frustrated to read. Understand that years of work that have gone into this project, that many many discussions about how things should be structured have been had, and that a ton of user feedback has been incorporated. Reading your comments it reminds me of the "Why don't you just" that Evan Czaplicki mentions in his great talk about the hard parts of open source. I recommend you watch it if you haven't seen it before.

All that to say, thanks for your feedback, but there might be a more constructive way to improve the Metals user experience than recommending the project to pause development and rearchitect its entire approach.

[vishalovercome]

Lesser emphasis on Java would be a good direction because then we can expect better Scala development experience (first class debugging experience, more refactorings, scala 3 features, etc.)

[tgodzik]

Thanks for opening the discussion!

Lesser emphasis on Java would be a good direction because then we can expect better Scala development experience (first class debugging experience, more refactorings, scala 3 features, etc.)

Actually, the main work around Java support was not done by the main contributors of the project, but rather independently by another contributor, who has done a tremendous job. I only added a small amount of polish afterwards. We are currently not working actively on Java support, but I might still consider some small additional features letter on. Despite what you are saying I found current Java support tremendously useful, especially for renames, implementation or references. Which were not showing the full picture before that.

As for a plugin ecosystem, that one would actually cause more work for the contributors without any assurances that other people would contribute plugins. Why would they contribute the plugins and not just the features to metals? It might actually be tougher to do with a plugin ecosystem, since it's really hard to come up with a good API and a bad one will cause more troubles than it's worth.

I have several ideas on how to improve the readability of the codebase, which might be useful, but I lacked the time for it unfortunately. Also, most of the actual complexity arise from the complexity of the working with a language.

Most of the support for different scala platforms should be working without issue, since the compiler provides us with a full information. There might be issues for scalapy since there are no actual types for python libraries, which is problematic. But in general most features depend on having the right classpath provided to the compiler.

If you have some concrete examples on what should be fixed in the current support then by all means let us know. There is a bunch of things beings worked on and improved, but i don't believe we need to change the architecture.

I do think we need to have post 1.0 roadmap, but I would not plan too much ahead in reality.

[vishalovercome]

It seems that the a few core points I made got lost amongst the various other things I've said.

1. Metals is free software which has helped me tremendously in my professional career. I have no problems with the effort. If anything, I'm thankful to all of you!
2. My core point is that Metals team may not be able to keep up with the various directions that the community is headed and would therefore like to know if there are ways in which community can share the development.
3. Being able to leverage Scala in frontend (scalajs), backend (java/scala) and data science (scalapy) would make Scala a unique language but that would only be possible if the tooling allowed for it to be possible. I agree with your point about limited usage of some of these projects. That's why I wanted to know whether each team, be it Scalapy users/Scala compiler team/whoever could somehow contribute more.
4. I specifically said that Metals team should provide some hints around the level of work required to support all the items I listed rather than switch development to something else. This again is to force community to contribute more rather than everybody expecting you to build things out eventually.
5. You would agree that number of feature requests and bugs are on the rise. This is promising in some sense as it indicates higher usage of Scala/Metals. However, would you not agree that this has to be dealt with differently?

Once again, thank you very much for your work!