Vision for Rust 2020 - Integration

With async close the completion everything looks to fall perfectly in place for maintaining the developer momentum that Rust has picked up. The language has surely shown to be approachable, flexible for new concepts and continues to be loved. So are we done? Of course not. I would like to see next years theme be Integration.

Language

One of the most productive features over its closest system language rival C is the wealth of ready-to-use libraries. However, with such close ties to its crate registry many projects are highly connected to Rust but to Rust alone. Only very few libraries come to mind that offer compatible C-API compilation modes. While this is partially explained by the usefulness and productivity of generics and traits it might also be indicative of holes in the language. One feature that might help here, existential types, has been in limbo for awhile.

Or how even dyn Trait seem incompatible with such an API. There is hope in that the problem is not impossible, for example see COM integration, but I'd rather look for a vendor independent solution. A full Rust ABI is too broad at this moment, I personally agree with the decision to close this RFC, but maybe we can solve the problem from the other way around. Maybe introduce #[repr(C)] for traits, with semantics being a this pointer and a struct of function pointers exactly as methods are defined in the trait, replacing &self with a void pointer. The pattern is arguable already popular in C but user data seldomly a separate parameter and more often a userdata member of a top-level struct (e.g. usage in fuse, sqlite, ...) This is mostly an idea since there are some details to RFC and figure out.

Tooling

Since we are looking at shared usage already, outside of crates.io, how is the situation for packagers? A quick look reveals that Debian has no policy for cdylib's in place at all, probably because they are not relevant.

Some of that is engrained in our tooling: cargo init can be used to setup standalone binaries or crates for publishing on crates.io but it is definitely not geared towards starting a C compatible crate, or a no_std crate. Even without going so far as to integrate cbindgen more deeply is there anything that can be done to facilitate this?

Overall, a focus on interaction with cargo, rustc from tools might be useful. Why was git so successful? Yes, surely being basically required for Linux kernel development got it attention but it's also straightforward to extend, script, and integrate once you understand its primitives. As a result, it is incredibly easy or at least well documented to convert many other VCS to git in a semi-automated fashion. What are the primitives of cargo or of rustc? Can we make them more accessible? Outside of IDEs?

I recently started building some better wrapper around cargo bench to persist carefully controlled measurements for historical comparisons similar to <perf.rust-lang.org>, but for any crate with #[bench] functions (It will be open sourced when ready. If You are interested or want to collaborate maybe sooner. Drop me a mail.) The basic commands are easily added but what if you adjust a benchmark and new data is not comparable to old measurements? Renaming only shifts the problem and is surely not an acceptable solution to all crate authors. One can imagine how quickly similar projects might also reach the end of their scripting wits and never come to fruition.

The compiler should have the desired information when compiling. That's maybe too specific but it would already help to get a file,line association for tests and benchmarks. rust-analyzer could likely find similar information but it is too focussed on LSP to be a generic CLI tool. Also, the main benchmark result can be parsed with nightly format=json output but it's already aggregated and hence missing a lot of relevant statistical data, for example the sample size! The compiler is great in its human interface but it feels too much like its built for direct or monolithic usage, and features only used in integration into larger systems seem to progress at a slower pace.

Community

When I think of full fletched programs with existing application outside the Rust ecosystem I can only name a few off the top of my head: ripgrep, alacritty, firecracker. Why is that? Why is a language that does offer unique benefits for sanely writing well performing parsers not more visible with conversion utilities ala awk, sed and networking tools such as curl, fetchamil? I don't expect anyone to have an immediate, complete answer to that question. But these are targets that can spark some continued usage outside the Rust developer community and as part of that attract programmers which might be interested in inspecting their tool's source code or even contribute back. If they grow to love the language like many of You already do, the better for all of us.

Integration includes wasm, of course. The ecosystem as a whole is naturally still young so this is not only Rust's responsibility. The llvm backend, available JIT compilers, debuggers, platform implementations, and not to forget the Specification itself are all in some form works in progress. This makes it exciting to contribute for sure but it also means that the workload might not be dominated by integration. An MVP exists so it might be more productive in the long run to wait for some dust to settle before making any hard to reverse commitments. Don't see this in conflict with the first argument, creating programs and tooling around wasm that just happens to be written in Rust would be awesome and could be a force multiplier.