Opencomputers: (Un?)managed memory

[SoniEx2]

I'm not sure what's what in terms of managed vs unmanaged so w/e.

Anyway, this memory would give you a raw block of memory and hopefully the OS (OpenOS) would provide an MMU.

Functions would include setInt getInt setFloat getFloat all the stuff you're used to and also compareAndSwap atomicAddInt etc.

We can then add real threads (well as real as running on a thread pool is, at least) which can access shared memory. This is where compareAndSwap and stuff comes in.

[payonel]

you already have shared memory, can do this with a custom table. _G is shared between processes and threads (event handlers and what not) already in openos. Thus you can

_G.shared_memory = {}

event.listen("key_down", function()
  if _G.shared_memory.halt then
    computer.shutdown()
  end
end)

-- then elsewhere, in another program
_G.shared_memory.halt = true

As for compareAndSwap and atomicAddInt, and CRITICAL_SECTION for that matter, these things have no purpose, no value, in a non-preemptive environment (where threads are purely cooperative). Which is what we have in OC with our lua archs

Otherwise, thanks for the suggerstion

[xarses]

slounds like a doc'er for #2686

[SoniEx2]

OC should be able to load a lua file/string into a new preemptive (or rather concurrent) thread.

[skyem123]

Actually... a block of memory with the ability to add and set bytes without taking up overhead would actually be useful, as a normal table to store strings or numbers has a lot of overhead, and a single string cannot be modified in place. While not standard to Lua, it would be helpful for stuff like emulators, and I'm sure there are other uses, too...

[purplemonday]

** note
so long you didnt specify it as local and in front of the thread.create it can already be used by other threads, there's no need to use _G namespace for this.
this also counts for coroutines etc
Only need for _G is when you need to expose it to literately everything to an even lower level bit of code and has such its risks in weirdness using it as the preferd method.

[DCNick3]

I agree with @skyem123. Of course you can write library, that will provide library for effective buffer manipulation by storing memory in strings, dividing it where it is accessed (break it down to small parts) and merging parts, that are not accessed, but it will still have enormous overhead, so this kind of feature will be very useful, In fact I'm working on some sort of bytecode interpreter and having this feature will reduce it's code size and complexity and increase speed.

[skyem123]

In some ways this is a Lua feature request 😛

[DCNick3]

In some way it's not a lua problem.. Lua already has ability to store arrays - tables, but it have big overhead (as a lua itself). OC have very limited resources, so why don't it have optimal ways to use them? I think, that this is not very had-to-implement feature and may event make pr... But will it be accepted?

[SaphireLattice]

If I understood this correctly, the feature requested is a second type of
memory that is pretty much byte array of fixed size, that, for Lua
architecture, exposes certain methods?

On Thu, 8 Feb 2018, 10:02 Nickie S., [e-mail] wrote:

In some way it's not a lua problem.. Lua already has ability to store
arrays - tables, but it have big overhead (as a lua itself). OC have very
limited resources, so why don't it have optimal ways to use them? I think,
that this is not very had-to-implement feature and may event make pr...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/MightyPirates/OpenComputers/issues/2693#issuecomment-363987600,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEFMuHPhBusqJ1p8KRKpkNfPRReuSHqaks5tSmOzgaJpZM4RKgst
.

[DCNick3]

@dangranos Yes, that's what have been in my mind. But, as I think, this should be more like C heap - you can allocate (and free) memory blocks of some (fixed) size. Something like
number heap.alloc(number size)
void heap.set8(number blockid, number offset, number value)
number heap.set8(number blockid, number offset)
void heap.set32(number blockid, number offset, number value)
number heap.get32(number blockid, number offset)
or perhaps use userdata for blockid and free it when GC occurs

[SoniEx2]

Definitely userdata. But why not support string.pack-style processing?

[DCNick3]

Does OC use lua5.3 as default? If yes - this would be definitely better than having tons of methods.

[DCNick3]

So, now we have something like
userdata heap.alloc(number size)
void block:free()
void block:set(number offset, string format, ...)
... block:get(number offset, string format)
(or
void heap.free(userdata block) (I think, that giving user control over deletion (not only by GC) gives more control and allows to write more stable programs)
void heap.set(number offset, userdata block, string format, ...)
... heap.get(number offset, userdata block, string format)
)

[SoniEx2]

No position?

[DCNick3]

Yep, lost that one, sorry

[skyem123]

Personally, I think that something like a mutable string would be more convenient than a "heap" based system.

[DCNick3]

Hmm, good idea in fact - it will be more flexible than heap and can perform same functions as heap.
For example you can build effective StringBuilder around that
Like
userdata mutstr.new(number sz)
str:replace(number offset, string or mutstr value[, number valueOffset[, number size]])
str:fill(char value[, number index[, number size]])
When you want to set binary values - you can use string.pack
But what about Lua 5.2? (string.pack was introduced in lua 5.3).
Another problem - when you use lua.pack it allocates immutable lua string that's not required in fact. It will increase GC load and so performance. Is it really good idea? How about embedding this feature to mutstr functions - it will solve both problems.

[SoniEx2]

And string.pack and stuff already have string embedding features.

[skyem123]

This reminds me of a class I wrote for an emulator (in Java). It allowed accessing the thing in different ways, big endian, little endian, byte, word, dword, qword... I guess having methods like those would be useful, and should be more efficient...

[DCNick3]

As @SoniEx2 already mentioned - string.pack have all of this features. Just implementing this kind of format syntax will give all of these features.
Also, as it's standard for lua, it will be easier for users to understand.
On the other hand - just passing string is not efficient at all. But it's just lua way - it's objective isn't efficiency, but embed-ability and easy understanding. Just look at type and file:seek functions. They're using strings at places where enum will better do, but it's not a lua way. I think, that we should provide maximum efficiency while keeping it simple and lua-like.

[skyem123]

Well... I'm thinking like...

local example = buffer.fromstring("example")
print(example:getbyte(0)) -- returns a string of 1 byte...
print(example:getwordle(0)) -- returns a little endian format of 2 bytes...
print(example:getwordbe(0)) -- big endian 2 bytes...
print(example:getsintbyte(0)) -- gets it as a signed integer...
print(example:getuintbyte(0)) -- unsigned!

...and so on? This is kinda silly but would be helpful for emulators

[DCNick3]

I understand what do you mean. We need to give It a think..
The string.pack-way

local b = buffer.fromstring("\1\2\3\4\5")
print(b:read("<b")) -- u8
print(b:read("<B")) -- s8
print(b:read("<i3")) -- s24
print(b:read(">i3")) -- s24, big endian

Probably it would be better to have two separate ways to do this - both of them have pros and cons

[SoniEx2]

Don't forget making a whole JVM or C emulator and precompiling classes/structs into pack/unpack strings.

[skyem123]

That's a bit extreme, the functionality can be implemented in Java/Scala interacting with Lua. Nothing else needed, and will still be useful.

[DCNick3]

After some investigations I found that calls to member of wrapped userdata (that one, that can be saved) creates tables (as table.pack), and overhead of creating a table is much bigger than string GC overhead, so there's no point in moving generation of actual byte sequence to write to write method. All methods, that @skyem123 mentioned can be implemented (efficiently enough) in usermode, so there's no point in increasing method count (I think so). As for now I'll stick with
strbuf.new(count: integer): userdata
buf:write(offset: integer, value: string)
buf:read(offset: integer[, length: integer]) (default length is 1)
buf:length(): integer (should it be a getter (without call)?)
buf:tostring(): string
API

The another questions is about this API being provided by separate component (like unmanaged memory controller) or being embedded into any Lua computer and thus sharing memory consumption with managed objects

[skyem123]

to be fair, the methods I mentioned were more for user convinience, no reason one could do it themselves, but it's not fun converting multiple bytes to an integer many times over!

[DCNick3]

Did you really read link about string.pack?
It's exactly what you need, but even more flexible - it converts strings to actual values (strings, floats, integers, structures of them). One problem is still here - string.pack was introduced in lua 5.3, but I think, that one can easily backport it to 5.2
In fact I have working code for it, but it yet ignores memory limits. I'll work on it for now.

[DCNick3]

like this

[skyem123]

neat! I guess backporting string.pack like functionality would be nice for this sort of thing...
(oh BTW, want to join the OC IRC channel? ^^;)

[DCNick3]

Meh, looks like memory limit is handled by C code, that is not part of OC. Well, this is bad. I don't like the idea of adding separate unmanaged memory component. Another option having separate unmanaged memory limit, that is also not really good. Third option is reducing max size of lua memory when allocation of unmanaged memory occurs, but it will require patches to computer.totalMemory. I will try sticking with third option for now

[DCNick3]

Looks like it works.
I will clean up code and do PR

[DCNick3]

So, PR is there.
Is 1.7.10 branch is OK?