Crates.io: Show Contents of Readme on Crate Pages

[killercup]

I consider the inline Readme file to be one of the most useful things on npm.

The Readme data is already available (and searchable, cf. #72), but I couldn't find any mention of a plan to show it on crate pages. As this can add a lot of content to a crate page (with possibly complicated markup), a new layout for the page might also be necessary.

[sfackler]

This'd be great.

[Luthaf]

Any new on this? How hard would it be to do?

[steveklabnik]

None yet. It shouldn't be _that_ hard, though it would require paying close attention to escapes.

NPM does this with markymark, I would suggest blatantly copying whatever they do.

[Luthaf]

NPM does this with markymark, I would suggest blatantly copying whatever they do.

Wouldn't that need a server-side node process? The current server is written in Rust, if I understand everything correctly.

[steveklabnik]

I didn't mean literally running their code; I meant investigating how they do so. What HTML do they escape? Do they show the whole README or a summary? Etc.

On Feb 19, 2016, 04:17 -0500, [email protected], wrote:

NPM does this with markymark, I would suggest blatantly copying whatever they do.

Wouldn't that need a server-side node process? The current server is written in Rust, if I understand everything correctly.

—
Reply to this email directly orview it on GitHub(https://github.com/rust-lang/crates.io/issues/81#issuecomment-186125695).

[Luthaf]

After some investigation, I managed to get a local version running, but when I try to upload a simple example crate it fails because it can not connect to S3. How can I do to load example data in a local website instance?

[steveklabnik]

It might be worth using the --proxy option and use the data from the actual
backend instead.

On Wed, Mar 2, 2016 at 4:45 PM, Luthaf [email protected] wrote:

After some investigation, I managed to get a local version running, but
when I try to upload a simple example crate it fails because it can not
connect to S3. How can I do to load example data in a local website
instance?

—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/crates.io/issues/81#issuecomment-191446706.

[Luthaf]

Yes, but I do not see how I can change the behaviour of the backend, to add an API point to get the README content as HTML in that case.

[steveklabnik]

Ah, sorry, I forgot about that part. I was only thinking about the frontend rendering aspect :(

[krzkaczor]

I have some time and willingness to work on this issue. @Luthaf can u help me with backend changes? Frontend changes seems to be trivial when we have nice endpoint returning readme as html :D

I couldn't find the project which is responsible actually for adding new packages into the repository. crates.io seems to be frontend and backend but it's only querying data - not adding them. I wanted to research textsearchable_index_col field in database because it seems to hold README contents and be used for full text search.

[Luthaf]

@Luthaf can u help me with backend changes?

Sorry, I have no knowledge at all of crates.io backend. I tried to do this, but gave up as I did not managed to get a local version of the backend working with some data.

[shssoichiro]

I'd like to step up and take a shot at implementing this.

[steveklabnik]

I mentioned this bug to @ashleygwilliams yesterday, and she mentioned that for @npm, they actually made this a separate service. One issue is that right now, crates.io displays stuff in the metadata, displaying READMEs would mean moving README stuff into the metadata, but that means that it blows up in size....

(By separate service I mean "implemented as a microservice", not a totally different website)

[ashleygwilliams]

:wave: hey all! i'm currently writing a blog post about the interesting history around dealing with READMEs for npm. right now, READMEs are part of the package metadata, and due to the potential for a DDoS, as well as the performance hit large READMEs cause, we've had to place an arbitrary restriction of 64kb on the READMEs. yes, yuck, arbitrary restrictions.

our goal with the READMEs in npm registry v3 will be to remove them from the package metadata completely and run them as a separate service that pre-renders the markdown and serves the generated html statically. this will remove any threat large READMEs would cause to performance or security and it will also have the added benefit of allowing us to render READMEs per version. it will also speed up our website since they will be pre-rendered and served statically.

i would greatly encourage ya'll to learn from our struggles and opt to keep the README _out_ of the package metadata. when ya'll become as popular as npm you'll thank me :)

if this is still open when the blogpost comes out later this week i'll be sure to post it here.

[jdm]

Blog post: http://blog.npmjs.org/post/153186129005/zero-one-infinity-readmes