Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2013-11-22

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 elyograg also, if you do a gluster volume status, does everything show online?
00:00 tyl0r in my fstab I put:
00:00 tyl0r [ip_addr]:/storage    /storage          nfs     defaults,_netdev,mountproto=tcp,vers=3,rw,bg,soft 0 0
00:00 tyl0r and then: mount /storage
00:02 tyl0r gluster peer status shows everything up
00:02 bennyturns joined #gluster
00:04 elyograg is there a /storage path on storage01 and/or storage02?  I'm just wondering if maybe it's mounting /storage (via the normal NFS server) rather than the gluster volume.
00:05 tyl0r No, but I'm using an older version of gluster (3.2.4) ... and the NFS client is really old (ubuntu 8.04)
00:05 elyograg I'm out of ideas.
00:05 tyl0r I know I need to upgrade 8.04 but it's not happening ATM. gluster should work on 8.04 but the client just hangs when I connect and ls the dir
00:06 tyl0r it's cool; thanks for your help
00:06 elyograg if you make a directory somewhere and do a FUSE mount of localhost:volume on your two gluster servers, does it have the right info in it?
00:07 elyograg mkdir /mnt/testmount ; mount -t glusterfs localhost:storage /mnt/testmount
00:08 tyl0r yeah, that looks good
00:09 elyograg i'm really confused, then.  you might try looking at the nfs log in /var/log/glusterfs (assuming 3.2 logs there, I only have experience with 3.3)
00:10 JoeJulian On your nfs mounted client: mkdir /storage/foo
00:11 tyl0r Okay, so when I umount and remount I can see the data on the NFS client
00:11 JoeJulian On your servers, ls -ld /export/brick*/foo
00:11 raghug joined #gluster
00:11 JoeJulian I thought you said stuff wasn't showing up on your bricks.
00:12 tyl0r It was only showing half of the data because of it being distributed.
00:12 JoeJulian A brick being /export/brickNN on your two servers.
00:13 d-fence joined #gluster
00:13 tyl0r okay, so that mkdir /storage/foo made it to all bricks...
00:13 JoeJulian Ok, cool.
00:13 tyl0r form the NFS client, I guess it's working. I will convince myself of that
00:13 JoeJulian hehe
00:13 tyl0r Thanks for your help guys
00:14 JoeJulian Maybe you were seeing the fscache...
00:16 tyl0r Oh gosh, it probably doesn't help that my system time was offset by 21389 seconds
00:16 tyl0r ... on nodeB
00:16 JoeJulian heh
00:16 tyl0r [nothing to see here]
00:28 dmueller joined #gluster
00:30 elyograg i was having a devil of a time copying data to my testbed volume.  turns out that the front USB ports (dta on USB external disks) suck.  One of them gave up the ghost entirely and the device disappeared.  put htem in usb ports on the back andit's much better now.
00:31 _pol_ joined #gluster
00:35 Technicool joined #gluster
00:36 kiwikrisp Any suggestions for RAM on the servers? I'm replicating ~12TB.
00:37 dneary joined #gluster
00:39 bennyturns joined #gluster
00:40 msolo joined #gluster
00:41 _pol joined #gluster
00:45 bennyturns joined #gluster
00:55 fidevo joined #gluster
00:56 B21956 joined #gluster
00:59 Shdwdrgn joined #gluster
01:03 mattappe_ joined #gluster
01:08 mattapp__ joined #gluster
01:10 sprachgenerator joined #gluster
01:13 mattapp__ joined #gluster
01:35 andreask joined #gluster
01:41 andreask joined #gluster
01:45 bala joined #gluster
01:46 y4m4 joined #gluster
01:56 _pol joined #gluster
02:02 guix joined #gluster
02:02 raghug joined #gluster
02:05 guix hi has anyone encountered the issue where fuse reports the message: inode not found?
02:06 guix I am using glusterfs 3.3.2
02:06 guix on centos 6.4
02:07 glusterbot New news from newglusterbugs: [Bug 1032894] spurious ENOENTs when using libgfapi <http://goo.gl/x7C8qJ> || [Bug 1033093] listFiles implementation doesn't return files in alphabetical order. <http://goo.gl/iIw5ci> || [Bug 1033275] The glusterfs-geo-replication RPM missing dependency on python-ctypes <http://goo.gl/fw8PV6> || [Bug 998967] gluster 3.4.0 ACL returning different results with entity-timeout=0 and without <http
02:07 JoeJulian guix: probably...
02:08 JoeJulian guix: What's not working?
02:08 _dist joined #gluster
02:08 guix well the glusterfs mount point using native glusterfs freezes and reports inode not found.
02:09 guix only way to bring it back was to unmount and remount the mount point
02:09 JoeJulian At the application layer, or just in the log?
02:09 guix this is while writing to the filesystem
02:10 guix this error message is found in the mount glusterfs logs
02:10 guix the gluster logs
02:11 JoeJulian From what I can see, that is only a warning so it shouldn't be the problem.
02:12 _dist evening everyone, I'm doing some throughput testing on a replica volume and I've noticed running vms have a tendency show up in "gluster volume heal vol info". It seems to be the case using native or api, is that normal?
02:12 JoeJulian _dist: yes
02:13 _dist JoeJulian: that's because of latency? It worried me at first, but everything seems to run fine despite it
02:13 guix well the things is the mount point is no longer acessible when this happens so I cat stat any file on the glusterfs
02:14 JoeJulian _dist: It's just a snapshot in time. Writes are pending... no big deal.
02:14 JoeJulian guix: what version?
02:15 guix JoeJulian: 3.3.2
02:16 _dist JoeJulian: cool, that's what I was hoping, just wanted to confirm it. I'll get back to my testing
02:19 JoeJulian guix: fpaste your client log
02:20 msolo left #gluster
02:21 guix JoeJulian: I don't have access to the log, as I am not at work. But what should I be looking for exactly?
02:21 guix Or is this the right place to troubleshoot this?
02:23 tyl0r joined #gluster
02:24 guix JoeJulian: I have to go, but thank you for answering. I will try to connect tomorrow.
02:30 harish_ joined #gluster
02:45 bharata-rao joined #gluster
02:51 _pol joined #gluster
02:55 sprachgenerator joined #gluster
03:02 SteveCooling joined #gluster
03:05 _BryanHm_ joined #gluster
03:26 shubhendu joined #gluster
03:27 sgowda joined #gluster
03:33 JoeJulian guix: Sorry, had to go too. Yes, this is a good place to hang out and troubleshoot.
03:38 mattapp__ joined #gluster
03:39 itisravi joined #gluster
03:44 _pol joined #gluster
03:47 bala joined #gluster
03:49 RameshN joined #gluster
03:50 bgpepi_ joined #gluster
03:55 mohankumar joined #gluster
03:55 mattapp__ joined #gluster
04:02 kanagaraj joined #gluster
04:31 kanagaraj joined #gluster
04:34 spandit joined #gluster
04:36 ndarshan joined #gluster
04:38 _pol joined #gluster
04:52 ppai joined #gluster
04:55 ababu joined #gluster
05:00 aravindavk joined #gluster
05:01 CheRi joined #gluster
05:07 vpshastry joined #gluster
05:09 dusmant joined #gluster
05:11 psharma joined #gluster
05:14 _pol joined #gluster
05:23 _dist left #gluster
05:30 satheesh joined #gluster
05:30 saurabh joined #gluster
05:44 lalatenduM joined #gluster
05:49 mohankumar joined #gluster
05:51 hagarth joined #gluster
05:51 nshaikh joined #gluster
05:56 kshlm joined #gluster
05:58 pravka joined #gluster
05:59 mohankumar joined #gluster
06:01 fidevo joined #gluster
06:13 shri_ joined #gluster
06:13 meghanam joined #gluster
06:13 meghanam_ joined #gluster
06:21 raghu joined #gluster
06:26 kevein joined #gluster
06:28 rastar joined #gluster
06:34 bulde joined #gluster
06:40 vimal joined #gluster
06:51 sgowda joined #gluster
07:02 ngoswami joined #gluster
07:03 shri_ hagarth: ping
07:04 shri_ hagarth: any luck for your devstack setup
07:12 _pol joined #gluster
07:19 _pol_ joined #gluster
07:28 jtux joined #gluster
07:37 harish_ joined #gluster
07:43 nshaikh joined #gluster
07:46 zerick joined #gluster
07:52 geewiz joined #gluster
07:53 sgowda joined #gluster
07:55 vshankar joined #gluster
08:00 ctria joined #gluster
08:04 ricky-ti1 joined #gluster
08:08 yinyin joined #gluster
08:12 hagarth shri_: I will have an update for you by Monday
08:13 eseyman joined #gluster
08:16 keytab joined #gluster
08:16 RameshN joined #gluster
08:19 shylesh joined #gluster
08:21 _pol joined #gluster
08:28 barnes joined #gluster
08:28 davidbierce joined #gluster
08:31 wica JoeJulian & semiosis: The ubuntu package of libvirt 1.1.4 for trusty (back ported it to saucy) works nice with glusterfs
08:31 * wica happy
08:38 samppah nice :)
08:38 vpshastry joined #gluster
08:39 Alpinist joined #gluster
08:43 andreask joined #gluster
08:44 wica If someone is intresested in running qmeu-1.5 with glusterfs support on ubuntu 13.10. https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517/comments/2
08:44 glusterbot <http://goo.gl/cNy36m> (at bugs.launchpad.net)
08:46 mgebbe_ joined #gluster
08:57 rastar joined #gluster
08:59 clarkee joined #gluster
09:02 shri_ hagarth: no problem.. I just ping... actually I'm facing problem with packstack on F19 ..as service sql throws error during install
09:02 shri_ hagarth: so I started debugging devstack.. nova commands
09:02 shri_ with pdb
09:06 calum_ joined #gluster
09:08 kshlm joined #gluster
09:16 tjikkun_work joined #gluster
09:21 hagarth shri: ok
09:22 andreask joined #gluster
09:22 ppai joined #gluster
09:26 shubhendu joined #gluster
09:35 CheRi joined #gluster
09:36 ProT-0-TypE joined #gluster
09:38 glusterbot New news from newglusterbugs: [Bug 1031328] Gluster man pages are out of date. <http://goo.gl/4e1PjH>
09:42 hybrid512 joined #gluster
09:45 ppai joined #gluster
09:51 muhh joined #gluster
09:52 raghug joined #gluster
09:55 samsamm joined #gluster
10:03 kshlm joined #gluster
10:03 kshlm joined #gluster
10:04 kl4m joined #gluster
10:07 bgpepi joined #gluster
10:08 nshaikh joined #gluster
10:09 raghug joined #gluster
10:20 harish_ joined #gluster
10:25 MiteshShah joined #gluster
10:38 glusterbot New news from newglusterbugs: [Bug 858732] glusterd does not start anymore on one node <http://goo.gl/X7NsZ>
10:43 dneary joined #gluster
10:49 franc joined #gluster
10:54 bharata-rao joined #gluster
11:07 mohankumar joined #gluster
11:14 shubhendu joined #gluster
11:17 bma joined #gluster
11:17 satheesh joined #gluster
11:21 krypto joined #gluster
11:29 meghanam joined #gluster
11:29 meghanam_ joined #gluster
11:36 vpshastry left #gluster
11:45 barnes Any recent performance tests on a glusterfs cluster with small files?
11:48 ricky-ti1 joined #gluster
11:55 barnes I'm thinking of a 4 node replica glusterfs with ~100,000,000 files and sizes 10-400K
11:58 rastar joined #gluster
12:02 meghanam joined #gluster
12:02 kanagaraj joined #gluster
12:10 rcheleguini joined #gluster
12:12 meghanam joined #gluster
12:12 calum_ joined #gluster
12:13 meghanam_ joined #gluster
12:17 glusterbot New news from resolvedglusterbugs: [Bug 765287] Remove ld path hardcoding for non 64bit systems. <http://goo.gl/ylcsFg>
12:23 RameshN joined #gluster
12:23 ndarshan joined #gluster
12:25 aravindavk joined #gluster
12:30 meghanam_ joined #gluster
12:31 chirino_m joined #gluster
12:39 glusterbot New news from newglusterbugs: [Bug 1033576] rm: cannot remove Directory not empty on path that should be clean already <http://goo.gl/tW3gtb>
12:47 vpshastry joined #gluster
12:50 raghug joined #gluster
12:50 LoudNoises joined #gluster
12:52 meghanam joined #gluster
12:52 psharma joined #gluster
12:56 vpshastry left #gluster
12:57 meghanam_ joined #gluster
12:58 KORG joined #gluster
12:59 satheesh joined #gluster
13:21 _BryanHm_ joined #gluster
13:22 ira joined #gluster
13:23 kl4m_ joined #gluster
13:28 partner joined #gluster
13:34 basic` joined #gluster
13:35 basic` joined #gluster
13:38 kl4m joined #gluster
13:44 meghanam_ joined #gluster
13:48 meghanam joined #gluster
13:54 bennyturns joined #gluster
13:56 davidbierce joined #gluster
13:56 eseyman joined #gluster
14:00 meghanam_ joined #gluster
14:00 mohankumar joined #gluster
14:11 hybrid512 joined #gluster
14:11 sroy_ joined #gluster
14:15 bugs_ joined #gluster
14:16 dbruhn joined #gluster
14:16 meghanam_ joined #gluster
14:19 meghanam joined #gluster
14:20 B21956 joined #gluster
14:21 B21956 joined #gluster
14:28 dbruhn Any of you guys who are typically adding hardware regularly using build scripts for your servers? or maybe using something like kickstar?
14:28 wica or foreman
14:29 wica But yes, here is one :)
14:29 dbruhn good to know
14:29 Norky joined #gluster
14:29 dbruhn How many systems are you managing wick?
14:30 wica depends on customer
14:30 wica from 20 to 700
14:30 dbruhn fair enough
14:30 hagarth joined #gluster
14:30 dbruhn Does forman handle the OS install and everything?
14:31 dbruhn why yes it does
14:31 dbruhn was looking at the wrong project
14:31 wica :)
14:31 wica foreman combines, preseed/kickstart and puppet with pxe boot
14:32 wica and a nice interface for the pple not like me
14:32 wica dbruhn: But why the question?
14:33 dbruhn Thats cool, that's mainly what I am combating, I have these systems that no one else knows how to add servers too, and it's fine now, but I like to play loose with my leash.
14:33 wica :)
14:33 bulde joined #gluster
14:34 dbruhn I have some interested parties who like to think they are technical in nature, but haven't gone beyond windows 2k3 and net app.
14:34 wica hehehe :)
14:35 dbruhn So it's more of a placebo to give them control of stuff to make them feel like they don't need me. I find that always makes my job easier.
14:35 wica true
14:37 mattappe_ joined #gluster
14:38 dbruhn 20-700 servers per system.... Are you one of the RDMA users out there?
14:39 wica remote direct memory access ?
14:40 dbruhn Yeah, instead of using tcp/ip for gluster
14:40 wica I'm a normal linux sysadmin. no special job title
14:40 dbruhn I thought you were saying you have gluster installs that are 20-700 servers
14:40 wica Noo, i'm using tcp and not all the 700 systems are used for glusterfs
14:40 dbruhn Infiniband, or 10GB?
14:40 wica The biggest gluster is 20 systems
14:40 dbruhn Ahh ok, makes sense.
14:41 dbruhn How much storage do you have in those 20 servers?
14:41 wica This setup 2Gbps per server
14:41 wica 50TB
14:42 dbruhn Fun, I was a bit jealous thinking about a 700 server gluster deploy out there, lol.
14:42 wica dbruhn: I don't think I would be happy then :)
14:42 dbruhn haha
14:43 bulde joined #gluster
14:43 wica me alone and 700 glusterfs servers + supporting servers
14:43 dbruhn Wouldn't be soooo bad!
14:43 wica Would like to do that...
14:44 dbruhn I think the worst gig I ever had for that kind of crap was 20 racks full up, 24x7 and I was the only admin.
14:44 meghanam joined #gluster
14:44 wica My max was around the 1400 servers in control, so this could work also
14:44 wica :)
14:45 micu2 joined #gluster
14:45 wica dbruhn: I know that :)
14:45 dbruhn It was pretty bad when my girlfriend at the time through my phone in the middle of the night after being woke up night after night from alarms.
14:46 dbruhn Out of data, out of support hardware and all
14:46 wica The systems are your babbies then, your girlfriend :)
14:47 wica My girlfriend as to eccapt that 1) computers....5) girlfriend :)
14:47 wica accept
14:49 failshell joined #gluster
14:50 bsaggy joined #gluster
14:54 l0uis joined #gluster
14:58 micu joined #gluster
14:59 vpshastry joined #gluster
15:03 vpshastry left #gluster
15:04 wushudoin joined #gluster
15:04 bulde joined #gluster
15:05 khushildep joined #gluster
15:05 micu2 joined #gluster
15:07 bgpepi joined #gluster
15:12 [o__o] joined #gluster
15:12 satheesh joined #gluster
15:14 meghanam joined #gluster
15:17 neofob joined #gluster
15:22 vince joined #gluster
15:26 hateya joined #gluster
15:27 meghanam joined #gluster
15:28 NeatBasis joined #gluster
15:29 NeatBasis joined #gluster
15:31 giannello joined #gluster
15:53 NeatBasis joined #gluster
16:00 khushildep joined #gluster
16:02 pravka joined #gluster
16:02 NeatBasis joined #gluster
16:03 meghanam joined #gluster
16:04 Guest19728 joined #gluster
16:06 Bullardo_ joined #gluster
16:08 raghug joined #gluster
16:11 meghanam joined #gluster
16:27 dkorzhevin joined #gluster
16:32 tomsve joined #gluster
16:35 sprachgenerator joined #gluster
16:43 muhh joined #gluster
16:48 vimal joined #gluster
16:49 rcheleguini joined #gluster
16:51 giannello joined #gluster
16:55 dbruhn wick, does dating you have a terms of service agreement at the beginning that they have to click Yes to accept?
16:56 dbruhn wica
17:00 bennyturns joined #gluster
17:04 aliguori joined #gluster
17:05 zerick joined #gluster
17:07 _pol joined #gluster
17:07 andreask joined #gluster
17:09 zaitcev joined #gluster
17:15 lpabon joined #gluster
17:23 raghug joined #gluster
17:25 meghanam joined #gluster
17:27 elyograg dbruhn: some people won't get married without a prenup, which I think is basically the same thing. :)
17:31 chjohnst_work joined #gluster
17:34 semiosis wica: glad to hear about the trusty package working.  thx for the info!
17:34 lpabon joined #gluster
17:35 ira joined #gluster
17:35 meghanam joined #gluster
17:37 lpabon_ joined #gluster
17:42 Mo__ joined #gluster
17:42 jbd1 joined #gluster
17:46 carmensarlo joined #gluster
18:09 the-me joined #gluster
18:10 glusterbot New news from newglusterbugs: [Bug 1033093] listFiles implementation doesn't return files in alphabetical order. <http://goo.gl/iIw5ci>
18:11 MacWinner joined #gluster
18:13 rotbeard joined #gluster
18:22 meghanam joined #gluster
18:37 coredump joined #gluster
18:39 coredump So I need some information about gluster network usage, is there any paper on it? Specifically I want to check what servers are involved on a file save operation. Like, if I am using striping and/or replication, when I save data the saving server sends data to multiple servers or to just one server (and then this one server redistributes the data)
18:39 dbruhn Are you using NFS or the Gluster FUSE client?
18:39 meghanam_ joined #gluster
18:41 coredump fuse
18:42 dbruhn The fuse client connects to each of the servers individually reading and writing from them without interaction to the other servers.
18:43 coredump so if I am getting 90 MB/s on dd writing to a fuse mount, means that the network between the saving server and one of the gluster servers is down to 90 MB/s
18:43 dbruhn So in a replication scenario the client writes to all replication targets simultaneously. Creating a need to bandwidth in a multiple of the number of replication processes being presented.
18:43 coredump meaning that a slow network limits operations
18:43 coredump right
18:44 dbruhn well keep in mind 90MB/s is actually 720mb/s and that sounds right about the high side of a 1gb network
18:44 coredump I can write to disks at 300 MB/s on the servers, but only at 90 MB/s using oflags=nocache,dsync (the kernel I am using doesn't support direct on fuse yet)
18:45 coredump dbruhn: I am running tests to conving my company that gluster is actually a good idea (over a expensive SAN)
18:45 dbruhn Totally understand.
18:46 robinr joined #gluster
18:46 coredump iperf gives me 112 MB/s (or 942 Mbits/s), and I am sure that they will say that gluster is being limited by something else and not by network (because network is 112 MB and dd is 90 MB/s)
18:46 coredump of course, there are differences on how those measurements are taken
18:46 coredump but still
18:46 coredump nitpicking
18:46 dbruhn Are you using a 1gb network?
18:47 coredump yes
18:47 dbruhn are the SAN's you are testing 1gbE or 1gb fiber?
18:47 dbruhn not really setting yourself up for an apples to apples if they are not
18:47 coredump but I intend to use a 10GB or at least a 8x1Gb on the final project.
18:47 StarBeast joined #gluster
18:47 coredump the SAN they want to get is 10Gb fiber
18:48 coredump but I can get 50% more storage by 50% less the price
18:48 coredump of course I can't beat a fiber SAN on IOPS/data transfer with gluster
18:48 coredump but for openstack cloud gluster will be more than enough
18:48 robinr hi, i have a volume that can't be started nor stopped; when i tried to start it says "already started" but ps auxwww | grep -i volume name reveals nothing. When I tried to stop it, it says "Stopping volume ULBpilot has been unsuccessful". Setting is replica = 2. Gluster version: 3.3.0-1. Is it safe to stop glusterd and delete /var/lib/glusterd/vols/VOLUMENAME as a way to delete volumes since it won't stop ?
18:49 dbruhn coredump, that bit of information makes a bit more sense. Actually gluster can perform quite well in that case. especially since v 3.4 has the addition QEMU support.
18:49 coredump yep
18:50 coredump I would love to show some use cases/papers if someone around here has some.
18:50 coredump specially benchmarks
18:50 MacWinner i'm running into a weird situation.. i have a gluster mount setup.. I can list the files in a directory fine.. but when I try to access the files, i get a "cannot stat <filename>: No such file or directory"
18:50 dbruhn 10GB ethernet is great, for the same price you can get into QDR infiniband and run 40GB with tcp/ip over it, as long as you don't need to get into a routable TCP/IP situation with your storage
18:51 MacWinner however ls is clearly showing the files there
18:52 dbruhn robinr, what does "gluster peer status" show, and what does "gluster volume status" show?
18:52 dbruhn MacWinner are you running a DHT volume? or Replication? or?
18:53 MacWinner dbruhn, replica 3.. will paste my command to set up the volume and mount
18:53 robinr gluster volume status revelas that the PID is that. https://dpaste.de/4IOO
18:53 glusterbot Title: dpaste.de: Snippet #248147 (at dpaste.de)
18:53 dbruhn use fpaste
18:54 dbruhn robinr, check your brick logs to see what kind of error is showing up. The bricks aren't starting for a reason.
18:54 robinr http://www.fpaste.org/56121/51464771/
18:54 glusterbot Title: #56121 Fedora Project Pastebin (at www.fpaste.org)
18:55 dbruhn robinr: hmm weird, so one peer is showing the bricks offline and one is showing them online?
18:55 robinr gluster volume info ULBpilot reveals that it is started. But, gluster volume status indicates that it is not true.
18:56 dbruhn robinr: check your brick logs as a start.
18:56 robinr mount shows that it's online for both.
18:57 robinr 0 size log  -rw------- 1 root root          0 Nov 22 13:30 mnt-gluster-ULBpilot-start.log, the other does not have that log at all.
18:58 MacWinner dbruhn: https://dpaste.de/mfxA
18:58 glusterbot Title: dpaste.de: Snippet #248148 (at dpaste.de)
18:58 MacWinner dbruhn,  this is output of various commands.. running latest gluster from yum repo.. 3.4
18:59 MacWinner dbruhn, i have 3 nodes in the cluster.. internal{3,4,5}
18:59 robinr no, i can't delete that gluster volume delete command via gluster
18:59 robinr because it says taht it's started
19:00 dbruhn MacWinner, have you tried unmounting and remounting?
19:00 MacWinner dbruhn, not yet.. but I can..
19:00 dbruhn robinr, have you tried restarting the server in question?
19:00 MacWinner dbruhn, just trying to figure out why it got in this state :)
19:00 dbruhn MacWinner, also are your client and server version numbers the same?
19:01 MacWinner dbruhn, yep, they are. 3.4.1-3
19:01 ipvelez joined #gluster
19:02 dbruhn macwinner, anything in your mnt logs when you are trying to ls those directories?
19:03 ipvelez hello good afternoon. I'm having issues trying to do "sudo gluster volume status"
19:03 dbruhn ipvelez, what is happening?
19:03 ipvelez it displays the following message: "Another transaction could be in progress. Please try again after sometime."
19:03 MacWinner dbruhn, yep.. just pasted the log errors into the dpaste
19:05 dbruhn macwinner, I am not seeing any update to the previous paste
19:05 MacWinner dbruhn: https://dpaste.de/gwOx#L2,3,4
19:05 glusterbot Title: dpaste.de: Snippet #248150 (at dpaste.de)
19:06 dbruhn macwinner, interesting it looks like it is having trouble setting the extended attributes, are the permissions on the bricks right for so the gluster daemon can modify the permissions?
19:06 meghanam_ joined #gluster
19:07 Technicool joined #gluster
19:08 MacWinner dbruhn, let me look more into that. This was a slightly strange scenereo because I had redone some of my setup so that I could switch the interfaces that my gluster cluster were connecting to.. i very well could have screwed something up with one of the setattr commands
19:08 dbruhn are you running SElinux? or what OS are you on?
19:08 MacWinner i thought it was working, but i guess I didn't actually test trying to access a file yesterday.. I just did an ls
19:09 MacWinner dbruhn, centos6.0.. no selinux
19:09 MacWinner setfattr -x trusted.glusterfs.volume-id $brick_path
19:09 MacWinner setfattr -x trusted.gfid $brick_path
19:09 MacWinner rm -rf $brick_path/.glusterfs
19:09 MacWinner oops.. sorry, I ran those commands yesterday to clear an error when I was tyring to create a volume for the bricks yesterday
19:11 ipvelez I have another question, what if I would like to delete everything related to gluster and start from scratch again, what should I do?
19:13 dbruhn macwinner, test system, not production right?
19:13 MacWinner dbruhn, yep :). was hoping to go into production soon
19:13 MacWinner dbruhn, i've decided to start fresh..
19:14 dbruhn macwinner, what file system are you using for the bricks?
19:14 MacWinner xfs
19:14 dbruhn kk
19:14 MacWinner i'm gonna re-fdisk and everything
19:15 dbruhn ipvalez, there is a hidden .glusterfs directory on the bricks, you need to remove that after you have deleted the volume.
19:16 dbruhn If you want to read up on some of the off things you'll run into JoeJulian has an awesome blog. http://joejulian.name/blog/category/glusterfs/
19:17 robinr joined #gluster
19:17 robinr dbruhn, not on selinux
19:17 robinr all my other volumes are working; except this one.
19:18 dbruhn robinr, can you fpaste the output of your df command on those servers
19:18 robinr dbruhn, http://www.fpaste.org/56131/85147920/
19:18 glusterbot Title: #56131 Fedora Project Pastebin (at www.fpaste.org)
19:20 MacWinner dbruhn, btw, before I redo mkfs on my drive, any tips you would give on modifying this command: "mkfs.xfs -i size=512 /dev/sdb1 -f"
19:20 dbruhn robinr, you're not getting logging information because your log fs is full
19:20 robinr argh..
19:20 robinr stupid me.
19:20 robinr my monitoring failed me and i failed my monitoring
19:21 ipvelez the problem is that when I try to delete the volume i get: "volume delete: my-test-volume failed: Another transaction could be in progress. Please try again after sometime."
19:21 dbruhn macwinner, that's all I use
19:21 MacWinner cool, thanks
19:24 robinr all rite; i'm a bit embarassed. it seems that /var/log/ being full is the problem.
19:24 robinr we've got monitoring setup; but somehow it didn't alert on us and thus we overlooked the /var/log
19:25 robinr thanks dbruhn
19:25 robinr really embarassed
19:25 dbruhn robinr, no worries, I have been bitten by that before
19:25 dbruhn You at least have /var/log on its on fs...
19:25 dbruhn I have some servers that/var is the partition.... lets just say gluster gets far more pissed off in that scenario
19:25 dbruhn robinr, is there anyway you can reboot the volume servers?
19:26 dbruhn i mean ipvelez
19:26 ipvelez yes, i could
19:26 ipvelez that woudl solve it?
19:26 dbruhn not sure, but it seems a process might be hung from that error
19:27 dbruhn robinr, the gluster logs can be a bit of a pain and they don't rotate properly.
19:27 dbruhn you might want to look at that
19:28 meghanam joined #gluster
19:28 dbruhn robinr, I suggest editing log rotate with copytruncate for those logs that don't rotate properly.
19:32 robinr yup
19:33 robinr thanks for the suggestion dbruhn, i'm also looking at our monitoring on why no alerts were sent out.
19:33 dbruhn No worries, hope it works out for you
19:35 TDJACR joined #gluster
19:36 meghanam_ joined #gluster
19:38 robinr logrotation settings were initially created by the glusterfs rpms. i think by default glusterfs configures to log 52 weeks of logs. with weekly rotation.
19:53 coredump carmensarlo: o/
19:53 dbruhn robinr, the problem is the default log setting need to kill the process to actually rotate, that's why you need to set copytruncate
19:54 dbruhn here is an example of the modification I have made to my logs
19:54 dbruhn http://fpaste.org/56141/15004713/
19:54 glusterbot Title: #56141 Fedora Project Pastebin (at fpaste.org)
19:55 meghanam joined #gluster
19:56 robinr thanks for sharing, dbruhn.
19:57 dbruhn I have had to use that because I have had logs run into the 16/20GB range
20:01 tomsve joined #gluster
20:02 khushildep joined #gluster
20:08 meghanam_ joined #gluster
20:23 geewiz joined #gluster
20:23 MacWinner when you run "mount.glusterfs internal3:/master /mnt/master"  on a bunch of servers, does this force each server to connect to internal3?  or will each server connect to a random node in the cluster which is discovered through internal3?
20:26 dbruhn macwinner, it will connect to the node specified, upon connecting to that node it gets all the peer information and then connects to all of the nodes
20:27 dbruhn If you need it to try other nodes it has been suggested by others to use rrdns as a method of having it loop through connections to the different nodes
20:31 MacWinner thanks!
20:31 dbruhn np
20:31 robinr joined #gluster
20:33 MacWinner dbruhn, does it follow any special algorithm to choose, or is it random?
20:33 MacWinner for the gfs mount i mean
20:33 dbruhn rrdns?
20:33 MacWinner dbruhn, no.. when you run mount.glusterfs
20:34 dbruhn the client actually connects to all of the servers in the cluster
20:34 MacWinner if I understood you correctly, the mount command will connect to the host specified, but then reconnect to another node?
20:34 dbruhn it then pulls and writes the data to the servers based off of the DHT/Replication/Stripe information
20:34 MacWinner ahh
20:34 MacWinner got it..
20:34 MacWinner so in essence, the client has the HA built into it as well..
20:34 dbruhn that's how it scales so well, is because there is really no centralization
20:34 dbruhn yep
20:35 MacWinner sweet
20:35 dbruhn so in a replication write, the client actually writes to all replicants at the same time
20:36 MacWinner dbruhn, is the distribution of the files done purely by filename?
20:36 dbruhn if I understand how DHT works, yes
20:37 dbruhn but if a storage location is full it will balance it off to another location
20:37 dbruhn and create a hardline
20:37 dbruhn hard link
20:38 dbruhn I'm a little bit fuzzy on how it works to be honest, every file stored in my file systems are numbers so... it just seems to work without issue for me
20:40 semiosis @mount server
20:40 glusterbot semiosis: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
20:41 semiosis MacWinner: the server specified in the mount command (when using -t glusterfs, FUSE) is used to retrieve the volume graph, which lists all the brick paths in the volume (server:/path) the client then connects directly to all those bricks, a tcp session for each
20:41 MacWinner gracias
20:42 semiosis as for placement, the filename is hashed & that hash is used to place the file on one of the distribution units (a brick or replica set)
20:42 meghanam_ joined #gluster
20:43 MacWinner semiosis, thanks, makes sense
20:43 semiosis yw
20:52 stopbit joined #gluster
20:58 MacWinner any ideas on how to try troubleshooting a very slow write speed.. I'm trying to rsync about 1200 files from a server into a mount point of a 3node gluster cluster..  i think this copy test went really fast yesterday.. today it's just very slow.
20:58 MacWinner i don't see any weird messages in the logs
20:58 dbruhn replica 3?
20:58 MacWinner and the files are coming in.. just very slow
20:58 MacWinner yeah, replica 3
20:59 dbruhn how slow are we talking?
20:59 dbruhn like 33MB a sec?
20:59 MacWinner like it should only take 1 minute max.. it's been like 15 so far
21:00 dbruhn couple things, small file operations on gluster are often times better suited with NFS
21:00 dbruhn second, every replication target means a multiple of bandwidth, replication is a bit expensive that way
21:00 MacWinner i did this same test yesterday and it was pretty fast.. only difference this time is that I have moved my gluster to use a second private network interface that I've dedicated for gluster.. i thought it would make things faster!
21:01 dbruhn hmm, is the client on the same network?>
21:01 MacWinner the node that is syncing the files is on a different network in a different datacenter.. i'm going to do a basic rsync test without copying to the gluster cluster
21:01 MacWinner just to baseline
21:02 dbruhn The reason I ask is because the peer information is returned to the client as it is from the servers.
21:04 MacWinner I just did a test of copying files from siteA to siteB1 with rsync.. its alredy done..  siteB has 3 nodes.. siteB1, siteB2, and siteB3 which form the gluster
21:04 dbruhn so the servers might be communicating on the private network, but the client that isn't able to access that private network is trying to connect to those private connection points
21:05 MacWinner when i copy from siteA into siteB1 into the folder that is mounted for gluster, it's just very very slow
21:05 neofob left #gluster
21:06 MacWinner dbruhn, the client is only running on the servers in siteB..  siteA is just doing an rsync so is not aware of gluster.. however when siteA rsyncs to /mnt/master, it's slow.. if it syncs to /rsynctest, it's fast.. (rsynctest is just a regular folder.  /mnt/master is a gluster mount)
21:07 DV__ joined #gluster
21:07 MacWinner also, if i ls /mnt/master on siteB1, it's much slower than if I do the same on siteB2 and siteB3
21:08 dbruhn Ok, so the glusterfs client is on the private network that the three gluster servers are on?
21:09 MacWinner correct
21:10 MacWinner i'm just trying to sync in data from another site
21:10 MacWinner that doesn't have gluster.. as part of my migration process
21:10 dbruhn what is the connection speed between the three gluster servers?
21:10 MacWinner 100mbs
21:10 dbruhn latency?
21:10 MacWinner the files are not big
21:10 dbruhn small file ops are actually super costly in a replication environment
21:10 MacWinner .8ms
21:11 daMaestro joined #gluster
21:11 dbruhn Ok, so one thing to think about is the GlusterFS client is connecting to all three servers, and then copying the data to each server, and waiting for a response.
21:12 dbruhn Small file ops with high latency are incredibly slow with any file system
21:13 MacWinner hmm.. i'm gonna try isolating it down a bit further.. i'll take remote rsync out of the picture and just copy from local directory into /mnt.. there very well could be something wrong with these private interfaces
21:15 MacWinner the rm is taking a while too
21:16 dbruhn since you are just testing, you might want to try a replica 2, and then a replica 3 as a test to, see the difference
21:16 dbruhn things like an ls even on a low latency system can be pretty slow
21:17 MacWinner dbruhn, even just doing the local copy test is really slow
21:17 MacWinner so wasnt' related to the remote rsync..
21:18 MacWinner i wonder if it has something to do with the public/private ip split on the servers
21:22 dbruhn gluster volume status <vol> clients might give you some insight on if your clients connected to all the servers
21:28 dbruhn john mark, emailed you earlier today, not sure if I am landing in your junk mail or not
21:29 dbruhn johnmark
21:41 bennyturns can I change my IP on my gluster nodes without re peer probing and re creating my volume?
21:43 MacWinner bennyturns, not sure.. but would be cool if it does work.. i'm sure you would need to have your dns updated
21:43 dbruhn did you add them via IP or via DNS?
21:43 bennyturns MacWinner, just a test system with some VMs, I don't have any DNS setup
21:43 bennyturns IP
21:45 dbruhn I believe you need to re-probe the servers then.
21:45 bennyturns kk
21:45 dbruhn the UUID of the server stays the same though, so it shouldn't have any issues with it thinking its a new server
21:46 dbruhn If you are really concerned backup your /var/lib/glusterd directory
21:49 dbruhn that is the dynamic configuration files for the system includes all of the volume information
21:51 bennyturns dbruhn, thks!  just doing some testing so not super concerned.  just gonna tear down and rebuild volumes/peers
21:51 dbruhn always good to understand the innards ;)
21:51 bennyturns yepo!
21:58 badone joined #gluster
22:03 primusinterpares joined #gluster
22:07 johnmark dbruhn: hey - I got it. Responding
22:07 dbruhn sweet
22:08 johnmark :)
22:14 JonnyNomad joined #gluster
22:18 davidbierce joined #gluster
22:25 MacWinner dbruhn, if I have 3 servers doing replica3 with /brick, would you suggest that I split /brick into /brick1 and /brick2 (each 500gb partitions on a single disk).. then do replica2 instead?
22:25 MacWinner so I'll have 6 bricks across 3 servers..
22:26 MacWinner dbruhn, btw, I just switched back to my public network interface and it seems like things are a lot faster.. so something going on when i use that second NIC
22:37 dbruhn macwinner, typically I build my bricks to facilitate my i/o needs and use RAID to increase IOPS.
22:38 dbruhn what is the purpose of replica 2 vs replica 3 for you?
22:38 MacWinner dbruhn, it just felt like it fit more naturally with my 3 servers initially
22:39 MacWinner i just needed backup since I don't have RAID
22:40 MacWinner if I do replica2 with 6 bricks, then I think I can get what i need in terms of redundancy, and not have the extra overhang of replica3
22:40 dbruhn Not to be one of those people, but RAID, and synchronous replication are not a replacement for backup.
22:40 MacWinner yeah, you're right..
22:40 dbruhn But, that being said. Replica 2 will provide you with failover abilities, and keep two copies of data.
22:40 MacWinner basically i just didn't want a single point of failure..
22:41 MacWinner dbruhn, the number of bricks has to be even multiple of the replica count right?
22:41 dbruhn then replica 2 will suffice
22:41 dbruhn yep
22:41 dbruhn with 6 bricks on 3 servers you will be able to do a replica 2
22:41 MacWinner k.. so I think maybe I'll just take the /dev/sdb drive and create 2 partitions
22:42 MacWinner dbruhn, will it automatically distribute the bricks across the servers?
22:42 MacWinner or the replica sets I mean
22:42 MacWinner i remember some blog post about rebalancing the bricks
22:45 dbruhn you need to put the brick in in order of replication
22:46 dbruhn are you ever planning on expanding the system beyond the 3 servers?
22:47 semiosis if you have 3 servers you should do replica 3
22:47 semiosis if you have 2 servers you should do replica 2
22:47 semiosis plain & simple
22:47 dbruhn semiosis, why wouldn't you want to do a replica 2 on three servers?
22:47 semiosis doing replica 3 on 2 servers is silly, doing replica 2 on 3 servers is crazy
22:48 semiosis dbruhn: asymmetry is unaesthetic
22:48 semiosis thats my reason
22:49 semiosis also, if you want to add bricks, it's complicated
22:49 dbruhn server1:/brick/A server2:/brick/B server2:/brick/C server3:/brick/D server3:/brick/E server1:/brick/F
22:49 dbruhn would work if he isn't expanding it ever
22:49 semiosis 640kb is all the ram anyone should ever need, right?
22:50 semiosis some smart guy said that once
22:50 dbruhn lol, well there is truth to that irony
22:50 dbruhn replica 3 is costly just for the sake of symmetry
22:51 semiosis meh
22:51 semiosis there's cost to having a more complex config that's harder to reason about
22:51 semiosis i saw something on the ML about this recently, let me see if i can find it
22:52 semiosis iirc the point was that the "best practice" recommendation is to have a multiple of your replica count of servers
22:53 dbruhn I don't disagree btw. Seems macwinner maybe was going between three sites with the data, but I could be wrong.
22:53 elyograg I prefer to use the same path for all bricks in my replica set with the same name, so dbruhn's example for me would be server1:/brick/A server2:/brick/A server2:/brick/B server3:/brick/B server3:/brick/C server1:/brick/C
22:53 semiosis note that this is a *requirement* for bricks, but a best practice recommendation for servers
22:54 MacWinner dbruhn, i do plan on expanding the server count
22:54 semiosis elyograg: yes indeed, it's recommended that if two bricks have the same content, they should have the same path (just on different servers)
22:54 elyograg i agree with semiosis, though.  if you have replica 2, add servers in pairs.
22:55 MacWinner k, i will just add another..  i have an odd number of servers because of my mysql+galera cluster setup..
22:55 dbruhn I agree as well, my systems are configured to be installed in pairs with a replica 2
22:58 gmcwhist_ joined #gluster
23:01 MacWinner what's the advantage of replica3?
23:02 MacWinner vs disadvanatge?  does it basically add 50% more latency?
23:02 semiosis it works well with quorum
23:02 MacWinner semiosis, ahhh.. that's why mysql+galera basically needs 3.. i ran into a sprit brain scenereo recently so I'm paranoid about it
23:02 semiosis the replica traffic is parallelized so latency should not grow linearly with more replicas, though it will grow somewhat
23:03 MacWinner got it, thanks
23:03 semiosis at least thats what i've heard
23:03 semiosis never tested that
23:04 badone joined #gluster
23:05 elyograg does anyone know the status of the arbiter node feature which should make quorum possible on replica 2 volumes?
23:06 semiosis i'm not familiar with that feature
23:07 semiosis when you use quorum with replica 2 your volume turns read-only when a brick is down (afaik)
23:08 elyograg http://review.gluster.org/#/c/4363/
23:08 glusterbot Title: Gerrit Code Review (at review.gluster.org)
23:10 elyograg I have four servers housing my replica 2 volumes.  I also have two peers that have no bricks.  Seems like I should be able to turn on quorum and have the volumes function perfectly if one of those six servers goes down.
23:12 semiosis elyograg: my guess would be that feature is stalled
23:13 semiosis elyograg: needs code review since feb, and the submitter is on vacation
23:21 glusterbot New news from resolvedglusterbugs: [Bug 820518] Issues with rebalance and self heal going simultanously <http://goo.gl/LUq7S>
23:32 andreask joined #gluster
23:34 jbd1 joined #gluster
23:39 andreask joined #gluster
23:40 bgpepi joined #gluster
23:44 fidevo joined #gluster
23:44 mattappe_ joined #gluster
23:50 raghug joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary