Finding Duplicate Records in MySQL

So I was needed to find some duplicate records in a table that had more than 3,000 records. I stumbled upon this query which made my life easier.


SELECT 'field', COUNT('field')
FROM 'table'
GROUP BY 'field'
HAVING ( COUNT('field') > 1 )

vi — ‘already locked, session is read-only’

I recently came across this error in Vi after I was editing a file and was disconnect: “foo.txt already locked, session is read-only.” I thought it was a swp file issue of some sort, so I immediately searched for it to remove it. Only to find out that one didn’t exist.

So I did what most people do and searched Google for the solution. I was lead to a message board and found out that since I was disconnected, Vi had an open process of some sort still ‘editing’ the same file. The solution was simple. All you do is run a ps and grep for vi and kill the process. Here’s the command I ran:

ps -ef | grep vi

Once you get the pid, just kill it with, uhh, kill.

Done!

Ternary; which means this stuff ? :

OK, I needed to write this down as I do not use this comparison on a daily basis, but use it enough where its getting annoying that I have to research it every time I use it.

So without further ado…..

( expression ) ? 'do this if true' : 'do this if false'

Voila

OOP in PHP not the same as Java

We all know that PHP5 has a lot more OOP features than PHP4, and we all know that it is not a real OOP language. There are a lot of things in PHP5 that won’t work like Java (obviously). With that said, I just want to point out one feature that I noticed while playing with PHP5 classes.

In Java, when you extend a class, it will run the parent’s class constructor first, then it will invoke the subclass’s constructor. Unfortunately in PHP5, when you extend a class, it will NOT invoke the parent class. You will have to manually invoke it yourself using the ‘parent::’ construct. Here’s some code (syntax may be wrong, excuse me, but you get the point):

class Hello
{
    public function __construct()
    {
      echo "Hello";
    }
}

class World extends Hello
{
    public function __construct()
    {
        parent::_construct(); //<-- must do that in order for the output to be "Hello World!";
        echo " World!";
    }
}

$test = new World();
?>

Output above would be..
Hello World!

If you didn’t add the parent::__construct(), output would be..
World!

Deny Users in a Directory and Allow Users in Subdirectory

It took me a while to figure this out. I wanted to deny the public access to a directory, like /deny and allow the public in a subdirectory like /deny/allow.

This was done by the specific order of Allow, Deny. If you mix these up, you will either allow all or deny all. Anyway, here’s what you do to deny users to /deny and allow users in /deny/allow.

<Directory /deny>
Order allow,deny
Allow from insert IP's
</Directory>

<Directory /deny/allow>
Order deny,allow
Allow from all
</Directory>

Stopping Starting Sendmail in FreeBSD

Here’s a quick reference for Sendmail users. The commands below are to start, stop, restart, and rebuild aliases for the Sendmail Daemon.

Stopping Sendmail
# cd /etc/mail
# make stop

Starting Sendmail
# cd /etc/mail
# make start

Restarting Sendmail
# cd /etc/mail
# make restart

Rebuild Aliases
# cd /etc/mail
# make

Enable Sendmail on startup
# echo ‘sendmail_enable=”YES”‘ >> /etc/rc.conf
# echo ‘sendmail_submit_enable=”YES”‘ >> /etc/rc.conf

Disable Sendmail on startup
# echo ‘sendmail_enable=”NO”‘ >> /etc/rc.conf
# echo ‘sendmail_submit_enable=”NO”‘ >> /etc/rc.conf

Done!!

Bash Test Operators

Here’s a list I have compiled that explains many Bash test operators. This is pretty much all I use. I think there’s more, but I will add to it as I discover them. I’ve been wanting to put this online so I can use as a reference and I finally did. So here it is, enjoy!!

FILES/DIRECTORIES
-e file exists
-f file is a regular file (not a directory or device file)
-s (lowercase ‘s’) file is not zero size
-S file is a socket
-d file is a directory
-b file is a block device (floppy, cdrom, etc.)
-c file is a character device (keyboard, modem, sound card, etc.)
-p file is a pipe
-h file is a symbolic link
-L file is a symbolic link
-t file (descriptor) is associated with a terminal device
-r file has read permission (for the user running the test)
-w file has write permission (for the user running the test)
-x file has execute permission (for the user running the test)
-g set-group-id (sgid) flag set on file or directory
-u set-user-id (suid) flag set on file
-k sticky bit set
-O you are owner of file
-G group-id of file same as yours
-N file modified since it was last read
f1 -nt f2 file f1 is newer than f2
f1 -ot f2 file f1 is older than f2
f1 -ef f2 files f1 and f2 are hard links to the same file
 
INTEGERS
-eq is equal to
-ne is not equal to
-gt is greater than
-ge is greater than or equal to
-lt is less than
-le is less than or equal to
< is less than (within double parentheses)
<= is less than or equal to (within double parentheses)
> is greater than (within double parentheses)
>= is greater than or equal to (within double parentheses)
 
STRINGS
= is equal to
== is equal to
 
The == comparison operator behaves differently within a double-brackets test than within single brackets.
[[ $a == z* ]] True if $a starts with an “z” (pattern matching).
[[ $a == “z*” ]] True if $a is equal to z* (literal matching).
 
[ $a == z* ] File globbing and word splitting take place.
[ “$a” == “z*” ] True if $a is equal to z* (literal matching).
 
!= is not equal to
< is less than, in ASCII alphabetical order
> is greater than, in ASCII alphabetical order
-n string is not “null.”
-z string is “null, ” that is, has zero length

Done!!

FreeBSD 6.2 + PHP5 + PHP5-extensions = php.core dumps!!

I had just recently installed 2 new servers and both of them had PHP dumps (php.core) and issued a segmented fault, signal 11. I installed PHP5 and installed PHP5-Extensions through the ports.

My first experience with the dump, I wrote a simple code to calculate interest rates, which was totally incorrect (my code btw). My second experience, on another server, was when I was installing pear-DB through the ports. It uses PHP to compile and it kept erroring and wouldn’t finish compiling. So I searched Google and found out that when you install PHP extensions via ports, these dumps occur. I am not sure about the exact reason why it dumps, but apparently PHP is very picky as to when the extensions are loaded, or rather the order in which they are loaded.

So my first search landed me to the FreeBSD lists, and there, they said to make sure the sessions module loaded first, or somewhat first. I edited my extensions.ini file (/usr/local/etc/php/extensions.ini) and moved ‘extension=session.so’ to the top of the list. I saved the file, and re-compiled pear-DB, and it still errored. So I researched some more and I was lead to pingle.org. His post mentioned the same extensions order as well, but placing some of the extensions last. He had recode, mysql, imap, sockets, and lastly pspell at the bottom of his list. I did the same and what do you know, no more dumps!! Woo Hoo.

From what people are saying is that when you have this “magic” order of extensions, to save the order because when you upgrade php or the extensions, it will overwrite this file.

Done!!

Areca + FreeBSD 6.2 + >2TB Installation

I’ve been trying to set up a backup file server using FreeBSD 6.2-RELEASE amd64, Areca RAID, and using a partition that is >2TB. My first install crashed the server under heavy writing. I was rsyncing 3 servers simultaneously. Checking the logs, there were some “kernel: g_vfs_done():…Error: 5” errors before the crash. I searched google and found out that it was a common error on Areca cards using FreeBSD. So I found a solution on the Areca website, http://areca.tw.com, and it said to edit the driver source code. I recompiled the kernel and started over again. This time, I am rsyncing 4 servers simultaneously and updating ALL the ports via cvsup (I had updated them earlier so there was nothing to really update. What I did was remove the ports tree, then I cvsup again). So far, no g_vfs_done errors and no crashes (knock on wood).

Here’s how I installed the server step by step:

1) I have 10x1TB Hitachi HD’s. I created 1 Volume at 100GB using RAID6 and another Volume at approximately 6.3TB using RAID6 as well. Make sure you use LBA 64 for any volume greater than 2TB. You can obviously create your own volumes depending on your usage. However, the 100GB Volume is just for FreeBSD.

2) Install FreeBSD on the first Volume, which should be da0. Do not touch the da1 volume.

3) When that’s done, install all the sources using sysinstall, and navigate to Configure > Distributions > src, and choose ALL.

4) Now we edit the driver. Open up /usr/src/sys/dev/arcmsr/arcmsr.c with any editor. You will see the following if statement at or around line 1304.

if(acb->srboutstandingcount >= ARCMSR_MAX_OUTSTANDING_CMD) {
pccb->ccb_h.status |= CAM_SCSI_BUSY; //<-- REMOVE THIS LINE
arcmsr_srb_complete(srb, 0);
return;
}

All you need to do is remove one line and add 2 more. The new if statement should look like this:

if(acb->srboutstandingcount >= ARCMSR_MAX_OUTSTANDING_CMD) {
pccb->ccb_h.status &= ~CAM_STATUS_MASK; //<-- ADD THIS LINE
pccb->ccb_h.status |= CAM_REQUEUE_REQ; //<-- ADD THIS LINE
arcmsr_srb_complete(srb, 0);
return;
}

5) Recompile the kernel and reboot (I was going to go into detail on the kernel recompile, but I will post this elsewhere).

Hopefully you have no errors up to this point. Now we partition the >2TB filesystem, mount it, and add to the fstab. From this point, I will use my volume set up. Just adjust to your settings if yours are different.

6) We will be using gpt to create a >2TB partition table.

# gpt create -f /dev/da1

7) Next, we add a gpt partition.

# gpt add /dev/da1

You can view the partition by using this:

# gpt show /dev/da1

8 ) Now we format the partition.

# newfs /dev/da1p1

9) Create a folder, let’s just name it /HUGE_STORAGE and then we mount it.

# mkdir /HUGE_STORAGE
# mount /dev/da1p1 /HUGE_STORAGE

10) Finally, we add this to fstab. Open up /etc/fstab and add this line to the bottom.

/dev/da1p1 /HUGE_STORAGE ufs rw 2 2

This exact set up has so far been a success. I will post anything that comes up, good or bad, here. Good luck!!

Done!!

UPDATE: Feb. 21, 2008 

I am happy to say that I have been monitoring this server and haven’t received any errors at all!! It runs about 3 rsyncs simultaneously nightly, and I’ve already tar/gzip’d  a 96GB file, mounted 2 samba mounts (which doesn’t have anything to do with the RAID card or hard drives), and moved/copied/removed/hard linked directories that were approximately 100Gb.