Ftl: First discussion

[technicalpyro]

based on the pihole script installing under the user "pihole" would it fit into the code that at the end of what im guessing will be the eventual bash/whiptail install script to have it run the following commands

sudo touch /etc/pihole/FTL.log
sudo touch /etc/pihole/FTL.pid
sudo touch /etc/pihole/FTL.lport
sudo chown pihole:pihole /etc/pihole/FTL.*

i would guess that is the eventual plan but dont want to assume

so far despite not running with a web interface as it is not agreeing at the moment every operation i have run has been successful and damn fast

[dschaper]

Yeah, the permissions and location of the files are most likely going to be updated as the engine takes shape. It's in preview release now for functionality, the details for other items will be tuned as we progress along. Web interface interaction is something that is a bit off, but the CLI directions should cover most of what you can do for now.

There is a log file and a script to generate a massive log file to see how performant the code is when accessing data that would crush an existing install, and the results are indeed fast!

Thanks for the information and comments and please post any other observations or ideas that you have and we'll handle them as we can.

[technicalpyro]

My current install is blocking just over 1.24 million domains and a command like >getallqueries which using the current web interface i really can't run due to the ajax error runs in seconds or less

couple features I would love to see:

an ability to view more than just the last blocked address ie >recentBlocked-x where x is the number of lines you want to view

the same could be said for getallquieries-x kind of an idea the time ability s nice but if you dont understand the timecode the database is using its difficult

knowing what the codes mean in the getallqueries results the numbers at the end of the lines would be nice as well or show it as a formatted table but that is definitely just a feature not a requirement

i understand right now this is alpha with me being guinea pig extraordinaire and everything that should be working is working
just me thinking out loud to be honest.

you guys rock and im happy to be allowed to play i really am :)

[DL6ER]

I like the idea >recentBlocked-x and >getallquieries-x and will add them.

in the getallqueries results the numbers at the end of the lines would be nice as well

I don't understand that?

The web interface implementation is finished but kept back for a while since we have to sort out how we do the licensing on that as it will integrate into the already published AdminLTE repo code

[technicalpyro]

in the getallqueries results the numbers at the end of the lines would be nice as well
I don't understand that?

1486179259 IPv4 api-v2launch.trakt.tv 172.16.1.107 2

the number two at the end of the line ( in this case) sometimes its a 1 or a 3 as well i just dont understand what it refers to

[PromoFaux]

just dont understand what it refers to

I have't actually read the FTL code properly yet, but I'm going to take a guess at the number being the pi-holed/OK status. I'm going to guess... 1=OK, 2=Pi-holed(exact), 3=Pi-holed(willdcard)

[technicalpyro]

That would make sense i had looked through but got lost in the code a few times

[DL6ER]

The statuses are assigned here:

• 0 = unknown, that should not happen (and has not happened to me so far), but could potentially happen - these lines will be written to the FTL log for possible subsequent investigation
• 1 = blocked by gravity.list (i.e. exact blacklisting)
• 2 = forwarded to upstream server
• 3 = answered by local cache
• 4 = wildcard blocking

[DL6ER]

or show it as a formatted table but that is definitely just a feature not a requirement

That is a difficult one since I don't know how long the domains and client host names can get, i.e. two unknown widths. If there would have been only one unknown width (like the domain name), I could have moved that to the end and could it have stay out of a preformatted table but I don't know how to do it as it is right now.

[technicalpyro]

yeah all the research I've tried to do tells me the limit is 255 bytes but that doesn't necessarily translate to x amount of characters

perhaps attempt local host names limit of 25 characters and see where it goes or just ignore it because it was a want not a need

[DL6ER]

yeah all the research ive tried to do tells e its 255 bytes but that doesnt necessarily translate to x amount of characters

Well, up to 255 characters

perhaps attempt local hostnames limit of 25 characters and see where it goes or just ignore it because it was a want not a need

I know about at least one user that has host names which are much longer than that because they reflect quite some hierarchy in the network, like:

device.room.building.field.company.tld

that can easily become > 25 characters

[technicalpyro]

Yeah for sure. i forgot about those who implement on the larger basis. My bad.

[technicalpyro]

installed the latest commits
change of filename structure to include pihole-FTL caused some issues all good now

[dschaper]

We've got to keep you on your toes! :)

[technicalpyro]

yeah and i learn from it too :)
Edit: tested >recentBlocked 10 works great

[DL6ER]

Oh, I wrote some text and somehow that seems to have gotten lost.

Yeah, you already discovered two changes:

• Filestructure (will change again to some more Linux-typical places like `/var/log/, etc. soon)
• >recentBlocked XY
• There is also >getallqueries (15). The ( ) have the nice effect that I was able to extend the scope to all already available filters, e.g.
  
  
  
  • >getallqueries (2) -> show only the recent two queries
  
  
  • >getallqueries-client 127.0.0.1 (10) show only the queries from localhost within the recent 10 queries (might return nothing)
  
  
  • >getallqueries-domain www.google.com (50)
  
  
  • etc.
  
  

Maybe I should put in the ( ) syntax everywhere for consistency?

[DL6ER]

The file locations have been changed to

/var/log/pihole-FTL.log
/var/run/pihole-FTL.pid
/var/run/pihole-FTL.port

You can use make && sudo make install to have the files created with proper permissions.

[technicalpyro]

Done this morning before work. will be back in approx three hours to reap the benefits. definitely like being able to control very easily the number of results returned very clean code:)

[DL6ER]

Yeah, but I'm not entirely satisfied by the new code, since e.g. >getallqueries-client 127.0.0.1 (10) will show only the queries from localhost within the recent 10 queries (might return nothing) instead of (what I'd expect to see) 10 queries from localhost.

The problem is that we print out the latest query as the end of the results, i.e. I don't know where to start printing.

[technicalpyro]

oh OK so it doent know to filter the last ten result containing that client ... you guys are t==he coding masters but will think about it while doing my shift today maybe i can think of something inelegant you guys can turn to gold

[technicalpyro]

might have to be a silent command kind of idea and i dont know how to code this but the workflow for me would look something like

• query database filtering for "domain" or "host" depending on what is specified
• print results to a temporary file
• query temporary file for number of results requested
• Print result to window
• delete temporary file

[dschaper]

Sounds like a good use case for a hashtable. (Coding a hash in C may not be trivial, I don't know that much yet...)

[DL6ER]

To be honest, I'm not sure if that is worth the effort. I'd have to do all the fancy filtering once, extract only what I need and loop over everything a second time, printing only what I need. This will not only slow down FTL but also (considerable) increase the memory needed for the request routines and make the code more complicated.

That is because I don't collect the data somewhere and print it only afterwards, but immediately push every matching data set out to the requester. This is one of the reasons why FTL is so fast.

[technicalpyro]

Will download and install release and report back when I get home ... Got called in for a stupidly long shift today

[technicalpyro]

release seem to install fine although despite running >kill to terminate before update when i went to start the service it echoed already running .. didnt realize it was started by the make && make install
not sure if it was implemented but >getallqueries 10 shows all not just the ten i thought it was limited to

[DL6ER]

didnt realize it was started by the make && make install

That does not happen. Either you had another instance of pihole-FTL running somewhere or we have a bug. Have you used sudo make install?

not sure if it was implemented but >getallqueries 10 shows all not just the ten i thought it was limited to

Use >getallqueries (10)

[technicalpyro]

ok did a clean install following same steps i did last night and i must not have killed the process like i thought everything is working well and installed as expected

also using the () with getallqueries worked perfectly

[technicalpyro]

god im slow some days got it sorted see other issue for info for some reason not letting me log in via CLI but got the right info where it needs to be

[technicalpyro]

thinking i might put osmc on a new card and see if we run into the same issues we are seeing on the master thoughts?
or is there a different OS i should be testing ?

[DL6ER]

Oh, I might have missed that - which issue?

[technicalpyro]

shit i accidentallyt referenced it in the issue can you delete?
1229 on main and i forgot the auto reference bullshit

[DL6ER]

No, the auto-reference cannot be deleted.

[technicalpyro]

goddamnit im sorry

[technicalpyro]

had the thought tonight when testing the recent change in blacklist. is there a plan to allow either a list or search function of the gravity list within the telnet instance or is that not needed due to the other places it can be done ?

[DL6ER]

Yeah, I think we won't add it right now.

Reason One: It is conveniently possible using either SSH (read and search on the files directly) and/or the web user

Reason Two: We don't read the files at all with FTL. I only count the number of lines for the statistics - the files are not accessed/parsed otherwise.

[technicalpyro]

both very sound reasons i completely agree

[technicalpyro]

i know it was mentioned above in this thread just wondering how the admin panel integration is coming along or what i should be doing to see if i can break that too ?

EDIT found the branch will be working my way through
EDIT 2 doing my head in im sure theres a simple step i missed. clone base repos and checked out the ftl branches but something didnt go right after i ran pihole -r ... not sure what but will just wait for it t be a little more done i use cli for most anyway

[DL6ER]

You will have to clone the branch from our AdminLTE repo.

Have a look at https://github.com/pi-hole/AdminLTE/pull/408

[technicalpyro]

took me some messing about but managed to get it working and i think i love you guys because DAMN this is fast
for reference the steps i took were as follows

• Downloaded base repos of both pihole and AdminLTE
• Checked ou new/FTL-binary and new/FTL-integration respectively
• ran pihole -r
• moved my executable copy of the binary into /etc/.pihole/
• started the FTL instance

something i nticed in all of this was the need to add the binary into /etc/.pihole/ I'm sure that was intended at this point and will be something not needed in future

either way i love it and you guys are freaking amazing

[technicalpyro]

tested this evening with the recent unofficial app for android and everything is looking great there too

[DL6ER]

something i nticed in all of this was the need to add the binary into /etc/.pihole/ I'm sure that was intended at this point and will be something not needed in future

I'm not sure why that was needed for you but the installation process is something that has to be finished. Looking into it over Karneval.

[technicalpyro]

when i tried using the new/FTL branch it failed when it went to compile FTL stating an error file not found on line 972 of the automated install.sh
when i ran pihole -r on the new/FTL-binary branch it appeared to go ok but didnt start the FTL daemon so i moved one of the binaries i had tested into /etc/.pihole/ and ran it manually and all working

[dschaper]

Can you detail out the step you took when you received the error and then the steps you took to resolve it in a new issue so we can track and fix the problem?

[technicalpyro]

done im pretty sure i messed something up i really hope so at least .... we will see if i can recreate it though too

[technicalpyro]

worth noting for the next commits that the following is missing an i in the README

Make it executable (chmod u+x phole-FTL*)

should be
Make it executable (chmod u+x pihole-FTL*)

[technicalpyro]

appears in the webadmin that loads are showing as 0 0 0 not sure if installing the latest release caused this but thats all that has changed on my rpi

[dschaper]

What does uptime command show on the Pi?

[technicalpyro]

one second running pihole -r to ensure latest commits are in use for each branch

[technicalpyro]

appears to be alright now after running pihole -r i would guess it was because i manually updated my binary this morning with the new release and that caused a minor break somewhere currently running
Pi-hole Version vDev (new/FTL-binary, v2.12.1-97-g87edbea) Web Interface Version vDev (new/FTL-integration, v2.4-93-gb54a276)

[dschaper]

Did the load averages go back up, or is a low load the proper reading?

[technicalpyro]

avereages are appearing accurate at the moment however and i dont know if this is intended are not "live" in the sense i ran uptime and got 13:42:53 up 3 days, 15:06, 2 users, load average: 0.40, 0.24, 0.27
but had to refresh the admin page to verify it matched

[dschaper]

Okay, those are decent numbers, have you been using the history and checking the web integration, there should be a marked increase in responsiveness.

[technicalpyro]

been doing a bit today but not a huge amount ... i just noticed the 0's the when i did run uptime didnt seem to match but refreshing the page helped

[dschaper]

Okay, thanks for helping out.

[technicalpyro]

looking at AdminLTE both before and after the latest commits talking about tailing the log ... the tail pihole log function does not seem to do anything. tailing on my device using pihole -t does show what i would expect to see

[DL6ER]

It should have have actually worked before and afterwards (but differently, I just pulled the necessary parts out of the API scripts, because this is not really where they should be located). Try the current version again after clear your cache. It works for me.

[technicalpyro]

ok will do and let you know

[technicalpyro]

no joy after clearing cache debug 1kzzzzq8jw I also noticed that if i try to update adlists from the web interface i do not see the output from running pihole -g in that page (thought i used to see that) and query adlists does not work for me either starting to think there might be a permissions issue somewhere

[DL6ER]

From your debug log, I see you are at commit b54a276 which is almost one week old (last Sunday). There was a bug fix in between by @PromoFaux .

Do you see any errors in the developer tools console in your browser (press F12)?

On the latest version of the AdminLTE branch you could have a look at

http://pi.hole/admin/scripts/pi-hole/php/tailLog.php

It should show something like

{"offset":1895443}

You can manually run gravity from this file:

http://pi.hole/admin/scripts/pi-hole/php/gravity.sh.php

and see if that works.

[technicalpyro]

hmm i though running pihole-r pulled the latest commit from the branch i had checked out let me get all of that up to date before i dive a little deeper then to your steps i shall go

[DL6ER]

Use pihole -up to update the branches.

[technicalpyro]

ahhhh thats where it got me i though the -r flag did both functions back into the code i go learning thanks for this

[technicalpyro]

looks like everything is sorted by the pihole -up to pull and install latest commits. the web tail didn't seem to respond until i ran pihole-t on the pi but that doesnt seem right to me however the browser side didn't populate until that command was run (using panes on my window box to track it)

EDIT the web tail stopped populating once i used [CTRL]+[C] to close the tailing on my device

[DL6ER]

Note sure if I get you. Does gravity update and adlists work now? Maybe you just have to wait long enough to see the tail log being filled? Try browsing the web parallel to having the taillog page opened.

[technicalpyro]

yes everything except the tailing is working correctly. the tailing seems to only be working if i have run and not closed the command pihole -t on my rpi. confirmed via parallel browsing with both CLI and web tail open while browsing

[technicalpyro]

and now all of a sudden it seems to be working properly

[DL6ER]

Javascript magic - you never know but some problems (that does not make sense) just vanish after some time

[technicalpyro]

makes me think of my job (electronics technician) i get near something broken it knows im there and i cant find anything wrong ... also makes me think of the old reddit thread talking about how to shoot yourself in the foot using all sorts of coding languages

[technicalpyro]

i see from the main repo FTL has been marked as testing required just wondering if there is anything else i should or could be doing

[DL6ER]

Before we go for the big main release, we want to test it at various places under regular and heavy workload to see if everything works as expected or if there are some rough edges that needs to be smoothened out. You are already helping with that and that is really appreciated!

Also we plan to install it on as many architectures as we can. In addition to the table you see in the README of the FTL repo, I successfully tried some virtual Debian, Ubuntu, and Fedora machines w/o any issues.

We will merge FTL into the main Pi-hole development branch as soon as we get the current release out (targeting this weekend) for enhanced testing.

[technicalpyro]

yeah only thing i havent done that i wanted to do for kicks was load every list in the wally3k list and see if i can break things

[DL6ER]

FTL does not need to load the blocking lists (except for counting the numbers of entries), so it will stay completely unnoticed

[technicalpyro]

ahhh was thinking it would stress test now i remember that wouldnt really bother it

[DL6ER]

I did once a stress test that was with 10 clients that continuously queries random character DNS records for several hours. In the end I had over 1 million entries in the query log and the web interface was still 100% usable.
The Raspberry died eventually because the log file grew too large for the 4GB Micro-SD card I have in there.

[technicalpyro]

there is always another bottleneck right lol

[technicalpyro]

noobie question time found this when looking at setupvars

INSTALL_WEB=true
INSTALL_WEB=true
INSTALL_WEB=true
INSTALL_WEB=true
PIHOLE_INTERFACE=eth0

its not breaking anything but seems odd that line is there multiple times i tried to figure out where that gets set but i haven't run the automated install script other than as pihole -r for repair at all since my original install

[dschaper]

Thanks, can you post that as a new issue on the Pi-hole repo? May be an installer bug.

[technicalpyro]

yupp wanted to ask here first ... feels safer to ask it and get a quick opinion first

[technicalpyro]

not sure its worth updating or not but installed flawlessly on my debian x64 VPS today if you wuld like to see the stats check out this link it is a closed resolver created using the openVPN setup recommended in the wikki

[DL6ER]

Thanks for sharing this with us.

I see on your dashboard that you have a

Memory usage: 49.0%

but that might come from the insane number of more than one million blocked domains.

[technicalpyro]

Also it's only a 512 MB memory size

On Feb 26, 2017, 11:47, at 11:47, DL6ER [e-mail] wrote:

Thanks for sharing this with us.

I see on your dashboard that you have a

Memory usage: 49.0%

but that might come from the insane number of more than one million
blocked domains.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/pi-hole/FTL/issues/1#issuecomment-282573151

[technicalpyro]

ps -aux has dnsmasq at 43% of memory. also probably due to something i missed when setting up the vps i dont show temp or FTL status in the dashboard and lastly and i will be submitting a bug report on this on the main repo the package dialog if it is missing breaks the install script the os provided by my host didnt include that so might be a good idea to include it in required packages that are loaded or looked for before installation

[DL6ER]

ps -aux has dnsmasq at 43% of memory

Okay, we can attribute that to the insane number of domains

dont show temp or FTL status

Temperature is not displayed if it cannot be detected - they does not work on most VMs (virtual machines have nothing like a "physical" temperature). Remember that there is actually no FTL status display except if FTL is offline.

[technicalpyro]

thats what i figured

[DL6ER]

FTL is now merged to Pi-hole development. Closing this very long issue now. Thanks for the very nice discussion. Feel free to open another discussion at any point!