LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > slarm64
User Name
Password
slarm64 This forum is for the discussion of slarm64.

Notices


Reply
  Search this Thread
Old 02-09-2023, 10:33 AM   #1
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
VLC-3.0.17.4


I'm on current-2023-01 with this and a few problems are showing with VLC
  • Ctrl_W doesn't work to close a file.
  • Occasional freezes.
I'll describe the freezes. The picture freezes, the keyboard/mouse is useless, and sometimes it's silent. The mouse moves, but keypresses are ineffective. At other times, anything up to a second of sound is repeated continuously: "I told you that might happen I told you that might happen I told you that might happen ....."

I'm going to watch stuff with an ssh shell open henceforth so I can try some remote diagnostics the next time it freezes. I'll post again when that happens. I have not much to go on unless someone has other experience of this?

Last edited by business_kid; 02-09-2023 at 10:35 AM.
 
Old 02-10-2023, 06:51 AM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Ok, I had vlc playing a video and htop watching things over ssh. The core usage was fluctuating on all cores 25-40%. When the video froze, so did htop. The ram and other values either were reasonable, or else not updated. I couldn't quit vlc or htop. The only solution was power cycling. I have reinstalled vlc, fscked / & tried a different kernel/modules pair. I watched that Video earlier in the week and VLC froze 3 times.

After the reboot, I restarted the same video in mpv. It played all the 90 minutes without incident. I can see from htop, and my wife was checking it. So it doesn't appear to be the video.

Here's the sha1 from my download:
Code:
bash-5.1$ sha1sum /mnt/hd/Arm/vlc-3.0.17.4-aarch64-1mara.txz
60adddd82d062d1856c5b51c03d5080e66bd34ad  /mnt/hd/Arm/vlc-3.0.17.4-aarch64-1mara.txz

Last edited by business_kid; 02-10-2023 at 08:00 AM.
 
Old 02-12-2023, 11:50 AM   #3
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Solved. I'm running at standard voltage & frequencies and it seems to work. It seems like a very obscure hardware failure maybe provoked by vlc.

The first time I played this latest test video through in vlc-3.0.17.4, it froze 3 times, although I got nearly 20 minutes out of my test video. Just because it fails once at a particular spot does not mean it will freeze there again. So I'm taking it the video is ok.

The 2 overvolts/frequency settings online that are touted online as working are "overvolts=2" & "arm_freq=1800" and "overvolts=6" & "arm_freq=2147." I've mainly been running on "overvolts=5" & "arm_freq=2000" for some time - this RazPi is 4 years old. Now it's tripping after 15 minutes or more on those settings. I tried "overvolts=4" and "overvolts=6" and they both died within 5 minutes. Overvolts=4 didn't even boot reliably. Lastly I tried "overvolts=6" & "arm_freq=2147" and that died too fast for me to ssh into it.

To put it in (British) Royal parlance, I am 'not amused' by this as my RazPi was gutless enough on software rendering without taking a 25% power cut into the bargain

So I'm left thinking:
  1. The Main SoC has developed some weakness provoked into a fault by overclocking.
  2. The power supply is dying. I think this unlikely as the voltage controlled by the overvolts setting is 1.2V or thereabouts. A variation at 5V should not matter.
  3. VLC, being the only program to date that triggers this, has some error.
 
Old 02-12-2023, 01:47 PM   #4
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
The power supply is dying. I think this unlikely as the voltage controlled by the overvolts setting is 1.2V or thereabouts. A variation at 5V should not matter.
Oh, but it does.

The lower voltages are generated internally by "buck regulators," power supply circuits that work by sending short pulses of current through an inductor (coil). If the supply voltage gets too low, the inductor will saturate and way too much current will pass through it. This is likely to damage the circuits receiving the current, as well as the transistors (MOSFETs) supplying it to the coil.

In fact, the easiest way to damage equipment that use buck regulators (typically laptops, routers, GPUs, and other equipment fed by an external, single-voltage power supply) is to use a PSU with a lower voltage rating.
 
Old 02-13-2023, 07:53 AM   #5
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Yes, I would call them switched mode power supplies. What you say about Input Voltage induced failure is not accurate for computr settings. Power supplies are blown regularly in Industry, usually by input spikes, or output short-circuits. All these 110/220V power supplies are switched mode (Or 'buck') regulators. These days they are a highly advanced science. Have you ever met a step-down for a cpu blown? I haven't.

The heart of the design is a switching mosfet and a fast diode from ground feeding an inductor, with a low ESR electrolytic capacitor on the other side. There's usually a comparitor controlling them, so a low input voltage means a slightly longer duty cycle. I'm sure you could design voltage tolerance out if you put your mind to it.

Last edited by business_kid; 02-13-2023 at 10:43 AM.
 
Old 02-13-2023, 02:36 PM   #6
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
Yes, I would call them switched mode power supplies.
Not quite the same thing. One can be considered a subset of the other.
Quote:
Originally Posted by business_kid View Post
What you say about Input Voltage induced failure is not accurate for computr settings.
Quote:
Originally Posted by business_kid View Post
Have you ever met a step-down for a cpu blown? I haven't.
That's only because emitting an abnormally low voltage is a really unusual failure mode for ATX-type PSUs: The 12V line is directly monitored by a comparator circuit responsible for generating the "Power Good" signal, and the buck regulator on the motherboard won't even attempt to start unless the PG signal is present. Same goes for GPUs; the onboard regulators don't even start if the supply voltage is too low.

However, take a look at any other type of equipment where it's possible to connect a PSU providing a low voltage, or where the PSU may provide a low voltage when failing. Blown high-side MOSFETs are quite common on laptops, and you regularly see damaged chipset components as well. Same goes for small routers and switches, where it's physically possible to connect the wrong type of external PSU.

Am I correct in assuming your Pi runs on a 5V USB supply, perhaps one of those cheap wall adapters? They can (and do) fail in all sorts of interesting ways, including supplying a low voltage.
Quote:
Originally Posted by business_kid View Post
The heart of the design is a switching mosfet and a fast diode from ground feeding an inductor, with a low ESR electrolytic capacitor on the other side.
Take a look at any recent equipment, and you'll find a single regulator chip, sometimes even with internal MOSFETs, feeding a coil and a capacitor. There will be no diode, as the circuit contains both high-side and low-side switching MOSFETs. The output voltage is fed back to a "sense" pin on the regulator chip via a simple voltage divider, where it's compared to an internally generated reference voltage (typically something like 0.6V).
Quote:
There's usually a comparitor controlling them, so a low input voltage means a slightly longer duty cycle.
Exactly. And since the coil has a really low inductance due to the high switching frequency, you know what will happen when the input voltage gets low enough that the duty cycle ends up extending beyond the saturation point of the coil.
Quote:
I'm sure you could design voltage tolerance out if you put your mind to it.
Why would an engineer design in low-voltage tolerance, when that would only mean increasing the BOM and result in business going to their competitor working in the office next door (in Shenzhen)?
 
Old 02-14-2023, 06:21 AM   #7
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
We both clearly know our hardware. Let's get on topic, and not waste energy refining each other's comments. Here's what I said:
Quote:
Originally Posted by business_kid
To put it in (British) Royal parlance, I am 'not amused' by this as my RazPi was gutless enough on software rendering without taking a 25% power cut into the bargain

So I'm left thinking:
The Main SoC has developed some weakness provoked into a fault by overclocking.
The power supply is dying. I think this unlikely as the voltage controlled by the overvolts setting is 1.2V or thereabouts. A variation at 5V should not matter.
VLC, being the only program to date that triggers this, has some error.
I'm inclined to discount the power supply as a source of failure, as I esteem the power supply to be better than the usual supplies, and lightly loaded into the bargain. I only have the RazPi, one SSD, one external wifi (≅100mA) and a keyboard & mouse receiver (<100mA) plugged in. If the supply was drooping low, increasing the overvolts setting should rescue my overclocking, which it doesn't. Presuming we eliminate the PSU, what's the most likely culprit?
 
Old 02-14-2023, 09:33 PM   #8
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
WI'm inclined to discount the power supply as a source of failure, as I esteem the power supply to be better than the usual supplies, and lightly loaded into the bargain. I only have the RazPi, one SSD, one external wifi (≅100mA) and a keyboard & mouse receiver (<100mA) plugged in.
The SSD will draw at least 1.2A, and perhaps as much as 2A under load. What's the amperage rating of the PSU?
Quote:
Originally Posted by business_kid View Post
If the supply was drooping low, increasing the overvolts setting should rescue my overclocking, which it doesn't.
If the PSU is drooping and/or there's switching noise/ripple on the 5V line, increasing the overvolt rating will also increase the current, making the problem worse.
Quote:
Originally Posted by business_kid View Post
Presuming we eliminate the PSU, what's the most likely culprit?
Assuming the PSU is good, and that the internal buck regulator is operating properly, and that there's adequate cooling, it's perfectly possible that this particular SoC just doesn't like being overclocked.

The reason I'm harping on about the PSU is that in my experience, power supply issues are the cause of something like 90% of all stability problems with computing equipment. Also, a PSU is pretty easy to eliminate as a possible source: If a scope or a multimeter isn't available (the latter won't show noise/ripple anyway), just try a different PSU, or reduce the load somehow and see if that has any effect.
 
Old 02-15-2023, 05:06 AM   #9
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Quote:
Originally Posted by Ser Olmy View Post
The SSD will draw at least 1.2A, and perhaps as much as 2A under load. What's the amperage rating of the PSU?
If the PSU is drooping and/or there's switching noise/ripple on the 5V line, increasing the overvolt rating will also increase the current, making the problem worse.
Assuming the PSU is good, and that the internal buck regulator is operating properly, and that there's adequate cooling, it's perfectly possible that this particular SoC just doesn't like being overclocked.

The reason I'm harping on about the PSU is that in my experience, power supply issues are the cause of something like 90% of all stability problems with computing equipment. Also, a PSU is pretty easy to eliminate as a possible source: If a scope or a multimeter isn't available (the latter won't show noise/ripple anyway), just try a different PSU, or reduce the load somehow and see if that has any effect.
  • I would agree with your suspicion of PSUs generally, but dismiss it in this case. I would also dismiss 90% as a figure for power faults. That's too high. My PSU is a replacement for the 'official' Pi PSU, which indeed sucked. It is a 5.1V 3A power supply bought as a replacement.
  • There is a big difference between 2.5" & 3.5" disks when you look at power consumption generally. The 2.5" ones are powered from the usb-3 port, rated at 0.5A. OTOH, 3.5" disks require a separate psu. You can't buy a case to house a 3.5" USB SSD. Mine is 2.5"

You seem to forget that this combination has sat here working on Slarm64 for years overclocked. Then I updated to current, updated to a new vlc, and that fault develops with the new VLC. The 3 Amps is rated for full usb current - 2.5A & 2×0.25A. I can change kernel, and play the video in MPV, but it crashes the cpu in VLC. It's quite total: ssh in apparently doesn't even receive an ACK, erroring out on "No route to host." [You can see why the PSU is unlikely; Overclocking is a slight flat extra current. The fault is not a spot in the video, and there's spare PSU capacity]. Only VLC causes errors - I can compile a Kernel & modules on 'make -j4' and it goes fine.

EDIT: Currently trying "overvolts=2" & "arm_freq=1800." First time I booted it, the cpu was going fine, but I had no video I cycled the power and it's running fine at 30-35% on each of the 4 cores. Presume it runs fine unless I come back with a second edit.

Last edited by business_kid; 02-15-2023 at 05:50 AM.
 
Old 02-15-2023, 06:10 AM   #10
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
My PSU is a replacement for the 'official' Pi PSU, which indeed sucked. It is a 5.1V 3A power supply bought as a replacement.
That should be more than sufficient.
Quote:
Originally Posted by business_kid View Post
There is a big difference between 2.5" & 3.5" disks when you look at power consumption generally.
I've never seen a 3.5" SSD. I was referring to 2.5" SATA drives. In fact, I got the 1.2A number from a stack of 2.5" SSDs I've got right here on the shelf.
Quote:
Originally Posted by business_kid View Post
The 2.5" ones are powered from the usb-3 port, rated at 0.5A.
USB3 ports are not limited to 500 mA, that's USB 2.0. All my old 2.5" USB enclosures come with those weird USB leads with two A plugs at one end, because only the very slowest 5400 RPM SATA drives draw 500 mA or less.
Quote:
Originally Posted by business_kid View Post
OTOH, 3.5" disks require a separate psu.
Because 3.5" drives require a 12V supply in addition to 5V, while 2.5" drives only use the 5V line. I don't know if this is part of some official standard, but it's been true for every 2.5" drive I've seen, mechanical as well as SSD.
Quote:
Originally Posted by business_kid View Post
You seem to forget that this combination has sat here working on Slarm64 for years overclocked. Then I updated to current, updated to a new vlc, and that fault develops with the new VLC.
And if you downgrade to an earlier version of VLC, does it start working again? If so, it's clearly a software-related issue. Perhaps later versions of VLC make use of hardware acceleration in a way that earlier versions couldn't, which in turn triggers the fault.
Quote:
Originally Posted by business_kid View Post
The 3 Amps is rated for full usb current - 2.5A & 2×0.25A. I can change kernel, and play the video in MPV, but it crashes the cpu in VLC. It's quite total: ssh in apparently doesn't even receive an ACK, erroring out on "No route to host."
That's a complete lock-up alright; not even ARP is running.

That's not something an application running under a non-root account should be able to cause, unless it's triggering an extremely serious kernel/driver bug, or there's a hardware issue.
Quote:
Originally Posted by business_kid View Post
Only VLC causes errors - I can compile a Kernel & modules on 'make -j4' and it goes fine.
That tells us it's related to the GPU part of the SoC, not the CPU cores.

Have you tried running a GPU benchmark? I wouldn't be at all surprised if that results in a lock-up as well.
 
Old 02-15-2023, 07:59 AM   #11
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Quote:
Originally Posted by Ser Olmy View Post
And if you downgrade to an earlier version of VLC, does it start working again? If so, it's clearly a software-related issue. Perhaps later versions of VLC make use of hardware acceleration in a way that earlier versions couldn't, which in turn triggers the fault.
I didn't try that. I had vlc-3.0.17.3, but deleted the backup yesterday . It's compiled against a different version of glibc, 2.33 vs 2.36


Quote:
Originally Posted by Ser Olmy View Post
That's a complete lock-up alright; not even ARP is running.
That's not something an application running under a non-root account should be able to cause, unless it's triggering an extremely serious kernel/driver bug, or there's a hardware issue.
That tells us it's related to the GPU part of the SoC, not the CPU cores.
Have you tried running a GPU benchmark? I wouldn't be at all surprised if that results in a lock-up as well.[/QUOTE]

Well, it holds the video on, repeats up to a second of sound, but locks up all right. It played fine at 1800Mhz. Now you see why I posted .

I haven't tried a GPU benchmark, as it's framebuffer/software rendering. Broadcom won't release their gpu driver, except to rpi os.
 
Old 02-15-2023, 08:20 AM   #12
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
I didn't try that. I had vlc-3.0.17.3, but deleted the backup yesterday . It's compiled against a different version of glibc, 2.33 vs 2.36
You could download the source and the SlackBuild script and build your own. It's a pretty straightforward procedure.
Quote:
Originally Posted by business_kid View Post
I haven't tried a GPU benchmark, as it's framebuffer/software rendering. Broadcom won't release their gpu driver, except to rpi os.
That's the part that doesn't make sense. Software rendering = high CPU load, low GPU load. The fact that you don't get a lock-up when compiling a kernel indicates that VLC is using the GPU, not the CPU.
 
Old 02-16-2023, 06:37 AM   #13
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
Quote:
Originally Posted by Ser Olmy
That's the part that doesn't make sense. Software rendering = high CPU load, low GPU load. The fact that you don't get a lock-up when compiling a kernel indicates that VLC is using the GPU, not the CPU.
When I went at the video end, I used the same .mkv test video segmnent for comparison. RPI OS could play it @≅60% (out of 400%) in top. But Update that RPi OS, and the sound goes, and stays gone.

Playing the same .mkv segment in Slarm64, even at 2 GHz, VLC tops out the cpu and won't render, whereas MPV is jerky, drops frames and the sound synch goes way out. It's a fair analysis that slarm64 is software rendering mkvs, yes? CPU usage is lower with .mp4s and lower again with .avi files. AFAIK only RPi OS & LibreElec (using RPi OS) can play that test video, presumably with Broadcom's driver. Maybe there's some element of gpu usage in Slarm64, but it certainly isn't full.

Now that the video situation is clearer, I'll restate the fault: My RazPi is locking up only while playing videos and overclocked to 2 Ghz in VLC-3.0.17.4. It seems to be running OK overclocked to 1.8Ghz. I think we can rule out power supply because moving the overvolts adjustment up or down at 2 Ghz makes things much worse. "Overvolts=2" & "arm_freq=1800" seems presently to be working but we'll see how long that will last.

Last edited by business_kid; 02-16-2023 at 12:13 PM.
 
Old 02-16-2023, 12:33 PM   #14
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,455

Original Poster
Rep: Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353Reputation: 2353
As suggested I built vlc(-3.0.18) and tested it. No change. Works @1.8GHz, locks up @2 GHz. I'm tiring of chasing after theoretical possibilities. That RazPi is mostly under 100% (out of 400%) load playing that video but locks up hard, as you pointed out.

It's also worth mentioning that overclocking the CPU does not overclock the GPU. There's a separate setting in /boot/config.txt for that, namely "gpu_freq=500" 500MHz is standard; I could but don't increase mine to 750-800MHz. Later EEProm revisions disable that feature unless you're on 4K video, and then limit it to 550MHz. But I'm keeping the familiar setup to avoid their attempts at UEFI.

The most interesting thing to come out of that was really sndwvs's automatically updating VERSION line in the SlackBuild which saves you farting about with editors every time they bump the software version.
Code:
NAME="vlc"
VERSION=${VERSION:-$(echo $NAME-*.tar.?z* | cut -d - -f2 | rev | cut -d . -f3- | rev)}
 
Old 02-17-2023, 10:36 PM   #15
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 3,347

Rep: Reputation: Disabled
Quote:
Originally Posted by business_kid View Post
As suggested I built vlc(-3.0.18) and tested it. No change. Works @1.8GHz, locks up @2 GHz.
So we're basically down to this: When overclocked, the system locks up under a specific type of load.
Quote:
Originally Posted by business_kid View Post
That RazPi is mostly under 100% (out of 400%) load playing that video but locks up hard, as you pointed out.
That is interesting, as it suggests VLC is using a single-threaded decoder.

That may also explain why you haven't been able to reproduce the issue by compiling a kernel: using four threads, no single core will see a continuous 100% load for an extended period of time, as it's periodically interrupted by disk I/O. You could try compiling using twice or three times as many threads as you have CPU cores, and see if the system remains stable.

It may still not crash, as I suspect that unlike a compiler, a video decoder will make extensive use of SIMD CPU instructions that activate a lot more circuitry.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
vlc 2.7 mageia 3 no video only sound vlc win ok moshebagelfresser Mageia 5 07-21-2013 12:20 AM
Installing VLC using vlc-0.9.9a-4.el5.rf.i386.rpm--Error relating to Dependencies redhat5 Linux - Newbie 1 12-17-2009 04:23 PM
LXer: Create your own VLC skin with VLC media player Skin Editor LXer Syndicated Linux News 0 12-06-2009 12:30 PM
VLC error: VLC could not open the file "/usr/share/vlc/skins2/text.bmp". brjoon1021 Ubuntu 1 01-14-2009 10:48 PM
I'm Getting 6 VLC Windows At Once - Problem With VLC Player davidx Linux - Software 1 11-03-2008 11:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > slarm64

All times are GMT -5. The time now is 10:10 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration