Crystal: .crystal in every working directory

[wrq]

Every time the crystal command is used, it dumps a bunch of compilation-related files to the working directory, essentially polluting every directory that contains crystal code.

Users have a /home directory that would be perfect for storing .crystal in, rather than the directory that I'm trying to work in.

I shouldn't have to .gitignore a new directory every time I want to write a new script.

Until this issue is fixed, I'm flabbergasted that people are even considering crystal for real-world usage.

[bcardiff]

You can set up in your profile export CRYSTAL_CACHE_DIR=~/.crystal so you choose where those generated file are stored and even reused.

[ozra]

Well, putting it into .gitignore is so simple I'm flabbergasted you can't do it ;-)

That said I'd also really like the option of setting a cache-dir in home-dir.

[ozra]

Oh, thanks @bcardiff how the fuck could I possibly miss that :-/

[bcardiff]

It's not easy to find, since the . is removed in github search so you can't search .crystal and get useful results. Here is the code related to that.

crystal init already adds /.crystal/ to .gitignore, so the users that builds apps or libs should not reach the annoyance of adding it :-)

[luislavena]

Shall we add this as subsection of _Using the Compiler_?

[wrq]

@bcardiff that should be the default behavior.

[waterlink]

To my knowledge, dumping everything to ~/.crystal may cause name conflicts in compile cache. Which would lead to unexpected weird behavior. That might be one reason it is in current directory by default. Could someone confirm my assumption?

Workaround is to somehow have it as ~/.crystal/<uniquely encoded current PWD>/. Then it will not have such problem.

[ozra]

Aha, I assumed the workdir-path was used like so, otherwise it's definitely problematic. (Haven't tried it out yet).

[luislavena]

With CRYSTAL_CACHE_DIR set, the result of the temporary files found in that directory are a little bit inconsistent:

Case 1: running crystal spec on a project

Given a clean .crystal directory, a new directory spec will be found inside, containing a flat representation of both Crystal files involved in the compilation and the source code of your project (a sample taken after running for luislavena/radix):

$ ls ~/.crystal/spec
Fiber.bc
File5858Stat.bc
File.bc
Float64.bc
GC.bc
Hash40123UInt644432Symbol1254432Bool41.bc
Hash40Int324432Ch-58936133be82407b4de468047ed7e7dc.bc
Hash40Int324432Process5858Status41.bc
...
Radix5858Node.bc
Radix5858Result.bc
Radix5858Tree5858DuplicateError.bc
Radix5858Tree5858SharedKeyError.bc
Radix5858Tree.bc
Random5858MT19937.bc
Random.bc

Contents of this directory are not nuked (removed) on every crystal spec running, so after executing the same for another project (like sdogruyol/kemal) total files grow from 348 to 610 files (both .bc and .o files). Since Kemal uses a different version of Radix, some of the Radix related files changed, but the ones generated from previous execution on different project weren't removed.

Scenario 2: compilation of a simple script

Given a simple bug.cr script, compilation using crystal build result in a total different structure, mapping the path up to where the script is found:

$ pwd
/home/luis/code/_sandbox

$ cat bug.cr 
require "http/client"

uri = URI.parse "https://google.com/"
client = HTTP::Client.new uri
response = client.get "/"

pp response.status_code
pp response.headers

This now results in the following structure created for that single script:

$ ls /home/luis/.crystal/home/luis/code/_sandbox/bug.cr
123Char4432Char4432Char4432Char125.bc
123UInt644432Symbol125.bc
40Fiber3245623240-1912b1087c9beb04149becc06298e7e2.bc
40Int3232456232Void41.bc
ArgumentError.bc
Array404032456232Nil4141.bc
Array4040Int3232456232Void4141.bc
Array40Int3241.bc
Array40Pointer40Void4141.bc
Array40String41.bc
AtExitHandlers.bc
Box40IO41.bc
...
LEBReader.bc
LibEvent25858EventFlags.bc
LibUnwind5858Action.bc
LibZ5858Error.bc
_main.bc
Math.bc
MemoryIO.bc
Nil.bc
OpenSSL5858BIO.bc
OpenSSL5858SSL5858Context.bc
OpenSSL5858SSL5858Socket.bc

A similar result is obtained when crystal run is performed.

So the only inconsistency on this seems to be crystal spec that could warrant ~/.crystal be set as default option.

I'm not sure if these issues have an impact on the generated code, but I think might help with consistency.

Cheers.

[ysbaddaden]

Thanks for the discovery. I was about to say that files were scoped by absolute path name in the .crystal cache, so there shouldn't be clashes. Maybe we can consider a bug that crystal spec doesn't respect that?

[will]

I put mine in /tmp/.crystal. Is there any reason not to make /tmp the default location instead of the current working directory (if you're on a system with /tmp)?

[refi64]

If you're working on a bunch of Crystal projects, using /tmp can be a bad idea. Namely, when you reboot your computer, Crystal will have to rebuild _everything_ again. In addition, putting too much stuff there really has the potential to eat up quite a bit of disk space; I had to set RPython's temporary directory to be on my external hard drive.

[jhass]

One argument against /tmp is that it usually is a ramdisk, I have mine in $HOME/.crystal, didn't clean for a while and well, it's 1.4G by now, which I wouldn't want to miss from my available memory.

One could do automatic pruning (delete oldest stuff until usage is below a certain threshold), but that adds complexity. And then what @kirbyfan64 said if you're among those who frequently reboot.

[luislavena]

@will I remember there was an issue when linking or attempting to run binaries across file system boundaries (/tmp on a different FS than your home directory) not to mention other permission issues.

[asterite]

I also find the .crystal directory a bit annoying, but it speeds up compilation of programs that you compiled at least once.

What we can do is:

1. Use $HOME/.crystal as the default directory, falling back to .crystal if $HOME doesn't exist
2. Only store 10 cache-related directories inside it, using a most recently used policy. Cache for each program goes into a separate directory inside the .crystal, and the cache is most useful when you are compiling a same program over and over with small changes. So having a most-recently-used policy is probably good.

That way we don't end up with .crystal directories everywhere, and the size of .crystal doesn't grow too much.

What do you think?

[will]

I think that's mostly good, but I'd make an exception for crystal eval so those don't evict 'real' programs.