Showing posts with label mysql. Show all posts
Showing posts with label mysql. Show all posts

2022/02/25

mecab-devel, where are you?

To compile MySQL from srpm on AlmaLinux8, you need mecab-devel, which doesn't seem to exist. After some digging around, this is the solution I found :

sudo yum --enablerepo=powertools group install "Development Tools"
sudo yum install make gcc-c++ rpmbuild

mkdir -pv ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
cd ~/rpmbuild/SOURCES
wget 'https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7cENtOXlicTFaRUE' -O mecab-0.996.tar.gz
cd ~/rpmbuild/SPECS
wget https://git.almalinux.org/rpms/mecab/raw/branch/c8-stream-8.0/SPECS/mecab.spec

rpmbuild -ba mecab.spec

cd ~/rpmbuild/RPMS/x86_64/
sudo yum install mecab*.rpm

This isn't perfect. Why would someone host their code on Google drive? But it seems this is what the author wanted.

2020/01/31

selinux vs searchd

If, like me, you use searchd and mysql then selinux will provoke the following message:

ERROR 1429 (HY000): Unable to connect to foreign data source: failed to connect to searchd (host=127.0.0.1, errno=13, port=931

The solution is simple:

semanage port -a -t mysqld_port_t -p tcp 9312

2019/10/30

Sphinx and Mysql 5.7

So I need to move my big application from CentOS 6 to CentOS 8. First problem - systemd. Unfortunately there's no way to remove systemd, and I'm not sure I want to move to Slackware. Second problem - mysql 5.6 won't compile with a recent GNU C++ compiler ! OK, I'll move to 5.7. Wait, SphinxSE won't compile with 5.7. It looks like Sphinx has been abandoned, but it's successor Manticore hasn't moved SphinxSE to 5.7 either. It turns out that 5.7 should have been called 6.0. Maybe this is why the next version is called 8.0?

Fortunately some Googling allowed me to find someone who did the hard work. I cleaned up compiler warnings and updated INSTALL. Source is here.

2017/05/01

daemontools, system V init and mysql

This is how you setup a babysitter for a service started with system V init scripts using DJB's deamontools. We can't just put service $service start into a run file, because sys V init scripts start up background daemons. We have to use the daemon's PID file to watch what's going on.

First, I create mysql-babysit. I'm using mysql as an example. For other services, adjust $service and $pidfile.

# mkdir /var/daemontools/supervised/mysql-babysit
# cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' > mysql-babysit
#!/bin/bash

service=mysql

datadir=/var/lib/mysql
pidfile=$datadir/$(hostname).pid


##################
sleepPID=
function sig_finish () {
    echo $(date) $service "$1"
    service $service stop
    [[ $sleepPID ]] && kill $sleepPID
}
trap 'sig_finish TERM' TERM
trap 'sig_finish KILL' KILL


##################
echo $(date) $service start

service $service start

if [[ -f $pidfile ]] ; then
    pid=$(< $pidfile)
    if [[ $pid ]] ; then
        while grep -q $service /proc/$pid/cmdline 2>/dev/null ; do
            sleep 60 & sleepPID=$!
            wait $sleepPID
        done
        echo $(date) $service exited
        exit 0
    fi
fi
echo $(date) $service failed to start
sleep 5
exit 3
SH

Next we create and activate the run script

cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' >run
#!/bin/bash

exec /var/daemontools/supervised/mysql-babysit/mysql-babysit
SH
# chmod +x run
# chkconfig mysql off
# service mysql stop
# cd ../../service
# ln -s ../supervised/mysql-babysit

We can control mysql with

svc -d /var/daemontools/supervised/mysql-babysit # shutdown mysql
svc -u /var/daemontools/supervised/mysql-babysit # startup mysql
killall mysql # restart mysql

2016/02/22

SELinux vs SphinxSE

It should be noted that SphinxSE wants to talk to searchd on port 9312. SELinux will prevent this. To enable it:

semanage port -a -t mysqld_port_t -p tcp 9312

2016/02/16

SELinux vs mysql

I'm a strange kind of fool. I maintain my own mysql packages, which makes installing them annoying because everything wants to pull in mysql-libs from the mainline.

I also sometimes want to install mysql in /home/mysql, not /var/lib/mysql as in standard on CentOS. SElinux is set up to prevent just this sort of thing. The short version is that everyhing in /home is has the home_root_t security context, which mysqld and mysqld_safe aren't allowed to interact with.

The solution is the following:

# first we are setting up the directory
mkdir -p /home/mysql/{InnoDB,etc,log,data,tmp,bin,sbin}
mv /etc/my.cnf /home/mysql/etc
ln -s /home/mysql/etc/my.cnf /etc
for n in /usr/bin/my* ; do ln -s $n /home/mysql/bin ; done
for n in /usr/sbin/my* ; do ln -s $n /home/mysql/sbin ; done
chmod 1777 /home/mysql/tmp
chown mysql:mysql -R /home/mysql
joe /home/mysql/etc/my.cnf  # change datadir
joe /etc/init.d/mysql       # change datadir and basedir

# now comes the part where we fight with selinux
semanage fcontext -a -t mysqld_db_t "/home/mysql(/.*)?"
semanage fcontext -a -t etc_t "/home/mysql/etc(/.*)?"
semanage fcontext -a -t bin_t "/home/mysql/bin(/.*)?"
semanage fcontext -a -t bin_t "/home/mysql/sbin(/.*)?"
semanage fcontext -a -t mysqld_tmp_t "/home/mysql/tmp(/.*)?"
semanage fcontext -a -t mysqld_safe_exec_t "/home/mysql/bin/mysqld_safe" 
restorecon -R -v /home/mysql
service mysql start

But it's still failing, because /home/mysql/bin/mysqld_safe is a symlink. To fix this, I did

grep mysqld /var/log/audit/audit.log | audit2allow -M "mysqlhome"
semodule -i mysqlhome.pp 
service mysql start

Yay! Now it works

2015/06/18

Still more fun with MySQL

Are you prepared to go mad? If so, compare these to statments and their results:

mysql> SELECT warehouse.NUM,warehouse.date FROM warehouse JOIN sphinx ON sphinx.id = warehouse.DID WHERE sphinx.query = 'filter=tid,288215463; index=Y2015,YD2015; limit=500; maxmatches=2000; mode=all; offset=0; query=dominique; sort=extended:date desc, sNUM desc' ORDER BY warehouse.date DESC,warehouse.NUM DESC LIMIT 50;
Empty set (0.00 sec)

mysql> SELECT warehouse.NUM,warehouse.date FROM warehouse JOIN sphinx ON sphinx.id = warehouse.DID WHERE sphinx.query = 'filter=tid,288215463; index=Y2015,YD2015; limit=500; maxmatches=2000; mode=all; offset=0; query=dominique; sort=extended:date desc, sNUM desc';
+---------+------------+
| NUM     | date       |
+---------+------------+
| AT00105 | 2015-06-17 |
+---------+------------+
1 row in set (0.00 sec)

What's going on is that MySQL is asking searchd (part of Sphinx) to do a full text search on 2 indexes. It then does a join on the results. With the ordering, I get zero results. Without ordering I get the expected results.

This shouldn't be happening. This can't be happening.

But then I found the answer: The first query in the example above was a cut and paste from the query log on my dev VM. This means that MySQL had already run that query and (more importantly) cached the results. The Sphinx indexes had been updated in the meantime. But searchd can't tell MySQL to invalidate the query cache.

mysql> SELECT SQL_NO_CACHE warehouse.NUM,warehouse.date FROM warehouse JOIN sphinx ON sphinx.id = warehouse.DID WHERE sphinx.query = 'filter=tid,288215463; index=Y2015,YD2015; limit=500; maxmatches=2000; mode=all; offset=0; query=dominique; sort=extended:date desc, sNUM desc' ORDER BY warehouse.date DESC,warehouse.NUM DESC LIMIT 50;
+---------+------------+
| NUM     | date       |
+---------+------------+
| AT00105 | 2015-06-17 |
+---------+------------+
1 row in set (0.00 sec)

Sanitiy is restored.

Long and short of this is to ALWAYS use SQL_NO_CACHE when using the Sphinx plugin.

2013/12/13

No fun

Trying to get Percona Server 5.6 to work with SphinxSE 2.1.2 was no fun. First off, it used to be that SphinxSE would create a load of .so .la, .a files. Now it just needs ha_sphinx.so. Second off, you have to execute INSTALL PLUGIN sphinx SONAME 'ha_sphinx.so'; This requires grant tables to be on (I generally have them off because it's less trouble and I have networking turned off in MySQL anyway, unless I have replication going). At least this is saved into mysql.plugin so I don't have to do it each time I restart mysqld.

And then, when I get it all set up, I get the following error

ERROR 1429 (HY000): Unable to connect to foreign data source: failed to resolve searchd host (name=localhost)

I'm WTFing just about as hard as I can right now. localhost is in /etc/hosts. It is also in my DNS server. Has MySQL gone insane? Percona? I'll blame Oracle, because they are just mean.

The solution, of course, is ALTER TABLE sphinx CONNECTION='sphinx://127.0.0.1:9312/YCUR';.

2012/03/18

Double joins

I finally groked something I was probably told to do 5 years ago. Actually I have no memory of being told to do this trick, but I should have been. It's so sweet. It's allowed me to rewrite and rethink the database queries for my archiving app.

If you don't know that a table join is, you'll have to learn about that first. So go do that and come back.

OK, now what is so sweet about a double join? First off, I'll point out that I'm not joining 3 tables together, I'm joining 2 tables together TWICE.

Why would I do this? A common idiom is to have a main table that contains all the fields that every widget (documents, articles in your store, whatever) has and another table that contains one or more rows per-widget that contain fields that only that widget or only that class of widget has. The alternative is to have one table that contains a field for every possible field of every possible class of widget, which can be very wasteful. And then you have to MODIFY TABLE each time you want to add or remove a field. Or you could have a table per class of widget, which reduces waste, but increases complexity.

So, a hypothetical ecommerce schema would looks roughly like:

CREATE TABLE main (
    int SKU PRIMARY KEY,
    timestamp created,
    int price,
    int in_stock,
    int category
);

CREATE TABLE extra {
    int SKU PRIMARY KEY,
    varchar(20) field,
    varchar(255) value
};

(Yes, extra.field really should be an ENUM or an INT.)

Now you want to find all the items with the keyword 'tablet', but you want to sort on the french name of the product:

SELECT SKU,E1.value as keyword,ES.value AS sort_field FROM main 
    JOIN extra AS E1 ON main.SKU = extra.SKU AND field = 'keyword'
    JOIN extra AS E2 ON main.SKU = extra.SKU AND field = 'fr'
    WHERE E1.value = 'tablet'
    ORDER BY sort_field;

Look at that, it's like you've temporarily added 2 new fields to main, 'keyword' and 'title'. What's more, to sort in English, I just change 'fr' to 'en'. To sort by manufacturer, change it to 'manufacture.'

Seems to me this invalidates 75% of the reason for NoSQL. The other 25% is that joins like the one above aren't very efficient for huge (aka web-scale) databases, the kind with multimillion records. The answer to that is that RAM IS CHEAP and USE SSDs.

2011/07/08

Fun with mysqldump

mysqldump makes for a quick and dirty way of backing up MySQL. Restore is trivial, if slow. However, it ties up the DB for the entire time it's running. Say you're like me and you have a table with 918,732,676 rows (yes, 9 hundred million rows). This means that the DB is tied up for 5 hours of dumping and 7 days for the restore. Clearly something better is needed.

However, before I work on integrating xtrabackup, I decided to see how slicing the dump with a LIMIT clause would work. mysqldump doesn't allow me to set LIMIT directly, however it does zero validation on the WHERE clause I can set. So:
(

mysqldump --opt DB --ignore-table BIGTABLE
mysqdump -NB --no-data DB BIGTABLE
MAX=$(mysql -NB DB -e 'select count(*) from BIGTABLE')
INC=$(( $MAX/1000 ))
seq -f '%.0f' 0 $INC $MAX | while read start ; do
mysqldump --compact --no-create-info DB BIGTABLE --where "1 LIMIT $start,$INC"
done
) > /my/backup.sql

This only works because mysqldump blindly tacks the WHERE clause onto it's select statement, giving us (roughly)
SELECT * FROM BIGTABLE WHERE 1 LIMIT 0,918732

This is a hack and a half: a- while it works now (Perconna 5.1.54) there's no guarantee it won't break at some point. But more importantly, b- this will still take a week to restore.

BTW, black-eyes to the idiots made %g the default seq format.

2011/02/02

CentOS4 and gcc4

Say you are merrily compiling someone else's code on CentOS4 and you get something resembling the following:
query_response_time.cc: In function `void query_response_time::add_time_atomic(query_response_time::TimeCounter*, uint64)':

query_response_time.cc:160: error: `__sync_fetch_and_add' was not declared in this scope
query_response_time.cc: In member function `void query_response_time::time_collector::collect(uint64)':
query_response_time.cc:261: error: `__sync_fetch_and_add' was not declared in this scope
make[3]: *** [query_response_time.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory /usr/src/redhat/BUILD/Percona-Server/sql'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory /usr/src/redhat/BUILD/Percona-Server/sql'
make[1]: *** [all] Error 2
make[1]: Leaving directory /usr/src/redhat/BUILD/Percona-Server/sql'
make: *** [all-recursive] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.49946 (%build)

You curse and swear. __sync_fetch_and_add is part of Intel's Itanium ABI and just plain isn't implemented in GCC 3, the default compiler in CentOS 4.

But the solution is easier then you think:
yum install gcc4 gcc4-c++

export CC=gcc4 CXX=c++4

OK, so maybe it's harder then that; you will probably have to add the following to your spec file:
BuildRequires: gcc4 gcc4-c++

%define cc_cmd ${CC:-gcc4}
%define cxx_cmd ${CCX:-g++4}

And replace any hardcoded reference to gcc or c++ with %{cc_cmd} or %{cxx_cmd}.