18 Jul
2014
18 Jul
'14
11:09 a.m.
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted for a brief
email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik queue) and they are
still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or is our partition of
Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority". Perhaps a
solution is increasing the priority of my jobs?
All the best,
Giangi
18 Jul
18 Jul
11:15 a.m.
Dear Gian,
I am having the exact same problem. I submitted a lot of short jobs and I
haven't ever had more than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
guerreschi(a)fas.harvard.edu> wrote:
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted
for a brief email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
queue) and they are still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or is
our partition of Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority".
Perhaps a solution is increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:17 a.m.
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com> wrote:
Dear Gian,
I am having the exact same problem. I submitted a lot of short jobs and I
haven't ever had more than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
guerreschi(a)fas.harvard.edu> wrote:
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted
for a brief email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
queue) and they are still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or is
our partition of Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority".
Perhaps a solution is increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:24 a.m.
I had this problem as well previously and specifying the correct amount of
memory and time (the least you need ;)) for the jobs solved the issue in
part, at least I have gotten ~100 jobs running a couple of times. Still it
is not very fast and sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com>
wrote:
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com>
wrote:
Dear Gian,
I am having the exact same problem. I submitted a lot of short jobs and
I haven't ever had more than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
guerreschi(a)fas.harvard.edu> wrote:
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted
for a brief email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
queue) and they are still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or is
our partition of Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority".
Perhaps a solution is increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:29 a.m.
I have a great little alias in my .bashrc that I feel might be appropriate
to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly prolific right
now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau <
stephanievalleau(a)gmail.com> wrote:
I had this problem as well previously and specifying
the correct amount of
memory and time (the least you need ;)) for the jobs solved the issue in
part, at least I have gotten ~100 jobs running a couple of times. Still it
is not very fast and sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com>
wrote:
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com>
wrote:
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
Dear Gian,
I am having the exact same problem. I submitted a lot of short jobs and
I haven't ever had more than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
guerreschi(a)fas.harvard.edu> wrote:
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted
for a brief email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
queue) and they are still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or
is our partition of Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority".
Perhaps a solution is increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:33 a.m.
Dear all,
This is partly my fault, because I have jobs running on 20 nodes that I
expect to run for quite some time (potentially months). My apologies.
However, there's also just an enormous backlog of jobs from many group
members. You can see the entire aspuru-guzik queue by doing:
squeue -p aspuru-guzik
This should give you an idea of why we are so backed up. As far as the
"Priority" tag you see after your jobs, this is a bit of a misnomer. The
jobs that say they aren't running because of priority are simply telling
you that they are lower priority than other jobs which aren't starting for
a different reason. That reason is "Resources", aka there are no available
nodes. Thus increasing your priority won't actually make a difference.
I will try and free up a few nodes, but things will likely remain backed
up.
Sam
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight <joey.goodknight(a)gmail.com
wrote:
> I have a great little alias in my .bashrc that I feel might be appropriate
> to share, here
>
> alias snoop="sacct -a -r aspuru-guzik -s r
> --format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
>
> It looks like the flow battery team is being particularly prolific right
> now?
>
>
> On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau <
> stephanievalleau(a)gmail.com wrote:
>
>> I had this problem as well previously and specifying the correct amount
>> of memory and time (the least you need ;)) for the jobs solved the issue in
>> part, at least I have gotten ~100 jobs running a couple of times. Still it
>> is not very fast and sometimes you need to wait a while.
>>
>>
>> On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com>
> wrote:
>>
>>> I have this problem now and had it before.
>>>
>>>
>>> On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com>
>> wrote:
>>>
>>>> Dear Gian,
>>>>
>>>> I am having the exact same problem. I submitted a lot of short jobs
>>>> and I haven't ever had more than two of them run simultaneously.
>>>>
>>>> Anyone else have the same problem?
>>>>
>>>> Jacob
>>>>
>>>>
>>>> On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
>>>> guerreschi(a)fas.harvard.edu
wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I forgot to discuss this topic in yesterday's group meeting, so
I
>>>>> opted for a brief email.
>>>>>
>>>>> I sent a couple of (pretty light) jobs in Odyssey (in the
Aspru-Guzik
>>>>> queue) and they are still pending since 1 week.
>>>>> Do you also experienced similar problems? Is there something wrong
or
>>>>> is our partition of Odyssey massively overloaded?
>>>>>
>>>>> Further info: In the reason of the pending it is written
"priority".
>>>>> Perhaps a solution is increasing the priority of my jobs?
>>>>>
>>>>> All the best,
>>>>> Giangi
>>>>>
>>>>> _____________________________________________
>>>>> Aspuru-list mailing list
>>>>> Aspuru-list(a)lists.fas.harvard.edu
>>>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>>>
>>>>>
>>>>
>>>> _____________________________________________
>>>> Aspuru-list mailing list
>>>> Aspuru-list(a)lists.fas.harvard.edu
>>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>>
>>>>
>>>
>>> _____________________________________________
>>> Aspuru-list mailing list
>>> Aspuru-list(a)lists.fas.harvard.edu
>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>
>>>
>>
>> _____________________________________________
>> Aspuru-list mailing list
>> Aspuru-list(a)lists.fas.harvard.edu
>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>
>>
>
> _____________________________________________
> Aspuru-list mailing list
> Aspuru-list(a)lists.fas.harvard.edu
> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>
>
11:40 a.m.
Before suleyman is thrown under the bus because he happens to have sent
many jobs recently, there are many of us who hit the queue hard at
different times. On the one hand, this is the nature of any shared
resource, sometimes it's under heavy load and wait times are long. On the
other hand, perhaps it's worth discussing options for the the lighter users
to have some nodes always available, or some kind of priority. And by
"discuss" I mean in a group meeting, in person, not in crazy email thread,
as fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight <joey.goodknight(a)gmail.com
wrote:
> I have a great little alias in my .bashrc that I feel might be appropriate
> to share, here
>
> alias snoop="sacct -a -r aspuru-guzik -s r
> --format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
>
> It looks like the flow battery team is being particularly prolific right
> now?
>
>
> On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau <
> stephanievalleau(a)gmail.com wrote:
>
>> I had this problem as well previously and specifying the correct amount
>> of memory and time (the least you need ;)) for the jobs solved the issue in
>> part, at least I have gotten ~100 jobs running a couple of times. Still it
>> is not very fast and sometimes you need to wait a while.
>>
>>
>> On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com>
> wrote:
>>
>>> I have this problem now and had it before.
>>>
>>>
>>> On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com>
>> wrote:
>>>
>>>> Dear Gian,
>>>>
>>>> I am having the exact same problem. I submitted a lot of short jobs
>>>> and I haven't ever had more than two of them run simultaneously.
>>>>
>>>> Anyone else have the same problem?
>>>>
>>>> Jacob
>>>>
>>>>
>>>> On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
>>>> guerreschi(a)fas.harvard.edu
wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I forgot to discuss this topic in yesterday's group meeting, so
I
>>>>> opted for a brief email.
>>>>>
>>>>> I sent a couple of (pretty light) jobs in Odyssey (in the
Aspru-Guzik
>>>>> queue) and they are still pending since 1 week.
>>>>> Do you also experienced similar problems? Is there something wrong
or
>>>>> is our partition of Odyssey massively overloaded?
>>>>>
>>>>> Further info: In the reason of the pending it is written
"priority".
>>>>> Perhaps a solution is increasing the priority of my jobs?
>>>>>
>>>>> All the best,
>>>>> Giangi
>>>>>
>>>>> _____________________________________________
>>>>> Aspuru-list mailing list
>>>>> Aspuru-list(a)lists.fas.harvard.edu
>>>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>>>
>>>>>
>>>>
>>>> _____________________________________________
>>>> Aspuru-list mailing list
>>>> Aspuru-list(a)lists.fas.harvard.edu
>>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>>
>>>>
>>>
>>> _____________________________________________
>>> Aspuru-list mailing list
>>> Aspuru-list(a)lists.fas.harvard.edu
>>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>>
>>>
>>
>> _____________________________________________
>> Aspuru-list mailing list
>> Aspuru-list(a)lists.fas.harvard.edu
>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>
>>
>
> _____________________________________________
> Aspuru-list mailing list
> Aspuru-list(a)lists.fas.harvard.edu
> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>
>
11:43 a.m.
Haha Tim :) Thanks for the heads-up.
I cleared all my pending jobs:
suleymaner@odyssey ~ $ squeue -u suleymaner |wc -l
141
, but still many pending from the group.
suleymaner@odyssey ~ $ squeue -p aspuru-guzik | grep " PD " | wc -l
2043
S.
On Jul 18, 2014, at 11:40 AM, Tim Hirzel wrote:
Before suleyman is thrown under the bus because he happens to have sent many jobs
recently, there are many of us who hit the queue hard at different times. On the one
hand, this is the nature of any shared resource, sometimes it's under heavy load and
wait times are long. On the other hand, perhaps it's worth discussing options for
the the lighter users to have some nodes always available, or some kind of priority. And
by "discuss" I mean in a group meeting, in person, not in crazy email thread, as
fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight
<joey.goodknight@gmail.com<mailto:joey.goodknight@gmail.com>> wrote:
I have a great little alias in my .bashrc that I feel might be appropriate to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly prolific right now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau
<stephanievalleau@gmail.com<mailto:stephanievalleau@gmail.com>> wrote:
I had this problem as well previously and specifying the correct amount of memory and time
(the least you need ;)) for the jobs solved the issue in part, at least I have gotten ~100
jobs running a couple of times. Still it is not very fast and sometimes you need to wait a
while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev
<zubarev.dy@gmail.com<mailto:zubarev.dy@gmail.com>> wrote:
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders
<jacosand@gmail.com<mailto:jacosand@gmail.com>> wrote:
Dear Gian,
I am having the exact same problem. I submitted a lot of short jobs and I haven't
ever had more than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian
<guerreschi@fas.harvard.edu<mailto:guerreschi@fas.harvard.edu>> wrote:
Dear all,
I forgot to discuss this topic in yesterday's group meeting, so I opted for a brief
email.
I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik queue) and they are
still pending since 1 week.
Do you also experienced similar problems? Is there something wrong or is our partition of
Odyssey massively overloaded?
Further info: In the reason of the pending it is written "priority". Perhaps a
solution is increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list@lists.fas.harvard.edu<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:46 a.m.
In theory fairshare(the job scheduling algorithm, not our personal job
rationing) should be taking care of this. In such systems, if you don't
run many jobs, the jobs you submit get super high priority and run
first. The last time it was this bad RC had secretly switched us to
"first come first serve". If they haven't this time, and it's still
an
issue, it may be worth having a conversation with them about rescaling
how priority is determined (it can usually be customized).
~Jarrod
On 7/18/2014 11:40 AM, Tim Hirzel wrote:
Before suleyman is thrown under the bus because he
happens to have
sent many jobs recently, there are many of us who hit the queue hard
at different times. On the one hand, this is the nature of any shared
resource, sometimes it's under heavy load and wait times are long.
On the other hand, perhaps it's worth discussing options for the the
lighter users to have some nodes always available, or some kind of
priority. And by "discuss" I mean in a group meeting, in person, not
in crazy email thread, as fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight
<joey.goodknight(a)gmail.com <mailto:joey.goodknight@gmail.com>> wrote:
I have a great little alias in my .bashrc that I feel might be
appropriate to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly prolific
right now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau
<stephanievalleau(a)gmail.com <mailto:stephanievalleau@gmail.com>>
wrote:
I had this problem as well previously and specifying the
correct amount of memory and time (the least you need ;)) for
the jobs solved the issue in part, at least I have gotten ~100
jobs running a couple of times. Still it is not very fast and
sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev
<zubarev.dy(a)gmail.com <mailto:zubarev.dy@gmail.com>> wrote:
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders
<jacosand(a)gmail.com <mailto:jacosand@gmail.com>> wrote:
Dear Gian,
I am having the exact same problem. I submitted a lot
of short jobs and I haven't ever had more than two of
them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian
<guerreschi(a)fas.harvard.edu
<mailto:guerreschi@fas.harvard.edu>> wrote:
Dear all,
I forgot to discuss this topic in yesterday's
group meeting, so I opted for a brief email.
I sent a couple of (pretty light) jobs in Odyssey
(in the Aspru-Guzik queue) and they are still
pending since 1 week.
Do you also experienced similar problems? Is there
something wrong or is our partition of Odyssey
massively overloaded?
Further info: In the reason of the pending it is
written "priority". Perhaps a solution is
increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
11:55 a.m.
I looked at the SLURM scheduler by using the command:
$ scontrol show config | grep SchedulerType
and the answer was:
SchedulerType = sched/backfill
which means (looking at some SLURM guide
https://computing.llnl.gov/linux/slurm/faq.html#pending) that:
If the scheduler type is builtin, then jobs will be executed in the order
of submission for a given partition. Even if resources
are available to
initiate your job immediately, it will be deferred until no previously
submitted job is pending. If the scheduler type is *backfill*, then *jobs
will generally be executed in the order of submission for a given partition
with one exception: later submitted jobs will be initiated early if doing
so does not delay the expected execution time of an earlier submitted job*.
In order for *backfill scheduling to be effective, users' jobs should
specify reasonable time limits*. If jobs do not specify time limits, then
all jobs will receive the same time limit (that associated with the
partition), and the ability to backfill schedule jobs will be limited. The
backfill scheduler does not alter job specifications of required or
excluded nodes, so jobs which specify nodes will substantially reduce the
effectiveness of backfill scheduling.
Here more informations:
https://computing.llnl.gov/linux/slurm/faq.html#backfill
Cheers!
S
2014-07-18 11:46 GMT-04:00 Jarrod <jarrod.mcc(a)gmail.com>om>:
In theory fairshare(the job scheduling algorithm, not
our personal job
rationing) should be taking care of this. In such systems, if you don't
run many jobs, the jobs you submit get super high priority and run first.
The last time it was this bad RC had secretly switched us to "first come
first serve". If they haven't this time, and it's still an issue, it may
be worth having a conversation with them about rescaling how priority is
determined (it can usually be customized).
~Jarrod
On 7/18/2014 11:40 AM, Tim Hirzel wrote:
Before suleyman is thrown under the bus because he happens to have sent
many jobs recently, there are many of us who hit the queue hard at
different times. On the one hand, this is the nature of any shared
resource, sometimes it's under heavy load and wait times are long. On the
other hand, perhaps it's worth discussing options for the the lighter users
to have some nodes always available, or some kind of priority. And by
"discuss" I mean in a group meeting, in person, not in crazy email thread,
as fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight <
joey.goodknight(a)gmail.com> wrote:
I have a great little alias in my .bashrc that I
feel might be
appropriate to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly prolific
right now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau <
stephanievalleau(a)gmail.com> wrote:
_____________________________________________
Aspuru-list mailing
listAspuru-list@lists.fas.harvard.eduhttps://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
I had this problem as well previously and
specifying the correct
amount of memory and time (the least you need ;)) for the jobs solved the
issue in part, at least I have gotten ~100 jobs running a couple of times.
Still it is not very fast and sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com>
wrote:
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com>
wrote:
> Dear Gian,
>
> I am having the exact same problem. I submitted a lot of short jobs
> and I haven't ever had more than two of them run simultaneously.
>
> Anyone else have the same problem?
>
> Jacob
>
>
> On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
> guerreschi(a)fas.harvard.edu> wrote:
>
>> Dear all,
>>
>> I forgot to discuss this topic in yesterday's group meeting, so I
>> opted for a brief email.
>>
>> I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
>> queue) and they are still pending since 1 week.
>> Do you also experienced similar problems? Is there something wrong or
>> is our partition of Odyssey massively overloaded?
>>
>> Further info: In the reason of the pending it is written "priority".
>> Perhaps a solution is increasing the priority of my jobs?
>>
>> All the best,
>> Giangi
>>
>> _____________________________________________
>> Aspuru-list mailing list
>> Aspuru-list(a)lists.fas.harvard.edu
>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>
>>
>
> _____________________________________________
> Aspuru-list mailing list
> Aspuru-list(a)lists.fas.harvard.edu
> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>
>
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
12:14 p.m.
Backfill is an extra functionality that lets small jobs slip in like
Stephanie mentioned. But the primary scheduler that determines what
jobs should be pulled from pending is what I mean, and the copied text
refers to a first in first out(FIFO) system. There are better systems
that account for how much you've run recently, how long the job is, how
much resources it takes up, how many jobs you have running currently etc
and the factors can (sometimes depending on the scheduler) be adjusted
to make everyone happier if it's a problem. Talking to RC about this is
probably the fastest way to get at what is currently being used and what
we can do about it.
On 7/18/2014 11:55 AM, Salvatore Mandrà wrote:
I looked at the SLURM scheduler by using the command:
$ scontrol show config | grep SchedulerType
and the answer was:
SchedulerType = sched/backfill
which means (looking at some SLURM guide
https://computing.llnl.gov/linux/slurm/faq.html#pending) that:
If the scheduler type is builtin, then jobs will be executed in
the order of submission for a given partition. Even if resources
are available to initiate your job immediately, it will be
deferred until no previously submitted job is pending. If the
scheduler type is *backfill*, then *jobs will generally be
executed in the order of submission for a given partition with one
exception: later submitted jobs will be initiated early if doing
so does not delay the expected execution time of an earlier
submitted job*. In order for *backfill scheduling to be effective,
users' jobs should specify reasonable time limits*. If jobs do not
specify time limits, then all jobs will receive the same time
limit (that associated with the partition), and the ability to
backfill schedule jobs will be limited. The backfill scheduler
does not alter job specifications of required or excluded nodes,
so jobs which specify nodes will substantially reduce the
effectiveness of backfill scheduling.
Here more informations:
https://computing.llnl.gov/linux/slurm/faq.html#backfill
Cheers!
S
2014-07-18 11:46 GMT-04:00 Jarrod <jarrod.mcc(a)gmail.com
<mailto:jarrod.mcc@gmail.com>>:
In theory fairshare(the job scheduling algorithm, not our personal
job rationing) should be taking care of this. In such systems, if
you don't run many jobs, the jobs you submit get super high
priority and run first. The last time it was this bad RC had
secretly switched us to "first come first serve". If they haven't
this time, and it's still an issue, it may be worth having a
conversation with them about rescaling how priority is determined
(it can usually be customized).
~Jarrod
On 7/18/2014 11:40 AM, Tim Hirzel wrote:
Before suleyman is thrown under the bus
because he happens to
have sent many jobs recently, there are many of us who hit the
queue hard at different times. On the one hand, this is the
nature of any shared resource, sometimes it's under heavy load
and wait times are long. On the other hand, perhaps it's worth
discussing options for the the lighter users to have some nodes
always available, or some kind of priority. And by "discuss" I
mean in a group meeting, in person, not in crazy email thread, as
fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight
<joey.goodknight(a)gmail.com <mailto:joey.goodknight@gmail.com>> wrote:
I have a great little alias in my .bashrc that I feel might
be appropriate to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly
prolific right now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau
<stephanievalleau(a)gmail.com
<mailto:stephanievalleau@gmail.com>> wrote:
I had this problem as well previously and specifying the
correct amount of memory and time (the least you need ;))
for the jobs solved the issue in part, at least I have
gotten ~100 jobs running a couple of times. Still it is
not very fast and sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev
<zubarev.dy(a)gmail.com <mailto:zubarev.dy@gmail.com>> wrote:
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders
<jacosand(a)gmail.com <mailto:jacosand@gmail.com>> wrote:
Dear Gian,
I am having the exact same problem. I submitted
a lot of short jobs and I haven't ever had more
than two of them run simultaneously.
Anyone else have the same problem?
Jacob
On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi,
Gian <guerreschi(a)fas.harvard.edu
<mailto:guerreschi@fas.harvard.edu>> wrote:
Dear all,
I forgot to discuss this topic in yesterday's
group meeting, so I opted for a brief email.
I sent a couple of (pretty light) jobs in
Odyssey (in the Aspru-Guzik queue) and they
are still pending since 1 week.
Do you also experienced similar problems? Is
there something wrong or is our partition of
Odyssey massively overloaded?
Further info: In the reason of the pending it
is written "priority". Perhaps a solution is
increasing the priority of my jobs?
All the best,
Giangi
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu <mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<mailto:Aspuru-list@lists.fas.harvard.edu>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
1:20 p.m.
Completely agree! We need fairshare. Sam: you can't hog nodes for months.
Do restarts so that your code checkpoints and reloads periodically and
others can fit in the queue in the meantime.
A) Please have a meeting with FASRC to discuss all of this ASAP
B) please ALL of you, there are unused cycles in our XSEDE allocation from
NSF that Suleyman manages. Don't be lazy and contact him for an account.
You can use the computer time too
Best
A
On Friday, July 18, 2014, Jarrod <jarrod.mcc(a)gmail.com> wrote:
In theory fairshare(the job scheduling algorithm, not
our personal job
rationing) should be taking care of this. In such systems, if you don't
run many jobs, the jobs you submit get super high priority and run first.
The last time it was this bad RC had secretly switched us to "first come
first serve". If they haven't this time, and it's still an issue, it may
be worth having a conversation with them about rescaling how priority is
determined (it can usually be customized).
~Jarrod
On 7/18/2014 11:40 AM, Tim Hirzel wrote:
Before suleyman is thrown under the bus because he happens to have sent
many jobs recently, there are many of us who hit the queue hard at
different times. On the one hand, this is the nature of any shared
resource, sometimes it's under heavy load and wait times are long. On the
other hand, perhaps it's worth discussing options for the the lighter users
to have some nodes always available, or some kind of priority. And by
"discuss" I mean in a group meeting, in person, not in crazy email thread,
as fun as these are. :)
-Tim
On Fri, Jul 18, 2014 at 11:29 AM, Joey Goodknight <
joey.goodknight(a)gmail.com
<javascript:_e(%7B%7D,'cvml','joey.goodknight@gmail.com');>>
wrote:
--
Alán Aspuru-Guzik | Professor of Chemistry and Chemical Biology
Harvard University | 12 Oxford Street, Room M113 | Cambridge, MA 02138
(617)-384-8188 | http://aspuru.chem.harvard.edu | http://about.me/aspuru
I have a great little alias in my .bashrc that I
feel might be
appropriate to share, here
alias snoop="sacct -a -r aspuru-guzik -s r
--format=User,JobName,Elapsed,AllocCPUs,AveCPU,NNodes"
It looks like the flow battery team is being particularly prolific
right now?
On Fri, Jul 18, 2014 at 11:24 AM, Stéphanie Valleau <
stephanievalleau(a)gmail.com
<javascript:_e(%7B%7D,'cvml','stephanievalleau@gmail.com');>>
wrote:
_____________________________________________
Aspuru-list mailing listAspuru-list(a)lists.fas.harvard.edu
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
I had this problem as well previously and
specifying the correct
amount of memory and time (the least you need ;)) for the jobs solved the
issue in part, at least I have gotten ~100 jobs running a couple of times.
Still it is not very fast and sometimes you need to wait a while.
On Fri, Jul 18, 2014 at 11:17 AM, Dmitry Zubarev <zubarev.dy(a)gmail.com
<javascript:_e(%7B%7D,'cvml','zubarev.dy@gmail.com');>>
wrote:
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
I have this problem now and had it before.
On Fri, Jul 18, 2014 at 11:15 AM, Jacob Sanders <jacosand(a)gmail.com
<javascript:_e(%7B%7D,'cvml','jacosand@gmail.com');>>
wrote:
> Dear Gian,
>
> I am having the exact same problem. I submitted a lot of short jobs
> and I haven't ever had more than two of them run simultaneously.
>
> Anyone else have the same problem?
>
> Jacob
>
>
> On Fri, Jul 18, 2014 at 11:09 AM, Guerreschi, Gian <
> guerreschi(a)fas.harvard.edu
>
<javascript:_e(%7B%7D,'cvml','guerreschi@fas.harvard.edu');>>
wrote:
>
>> Dear all,
>>
>> I forgot to discuss this topic in yesterday's group meeting, so I
>> opted for a brief email.
>>
>> I sent a couple of (pretty light) jobs in Odyssey (in the Aspru-Guzik
>> queue) and they are still pending since 1 week.
>> Do you also experienced similar problems? Is there something wrong or
>> is our partition of Odyssey massively overloaded?
>>
>> Further info: In the reason of the pending it is written "priority".
>> Perhaps a solution is increasing the priority of my jobs?
>>
>> All the best,
>> Giangi
>>
>> _____________________________________________
>> Aspuru-list mailing list
>> Aspuru-list(a)lists.fas.harvard.edu
>>
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>
>> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>>
>>
>
> _____________________________________________
> Aspuru-list mailing list
> Aspuru-list(a)lists.fas.harvard.edu
>
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>
> https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
>
>
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
_____________________________________________
Aspuru-list mailing list
Aspuru-list(a)lists.fas.harvard.edu
<javascript:_e(%7B%7D,'cvml','Aspuru-list@lists.fas.harvard.edu');>
https://lists.fas.harvard.edu/mailman/listinfo/aspuru-list
3592
days inactive
3592
days old
aspuru-list@lists.fas.harvard.edu
11 comments
11 participants
participants (11)
-
Alan Aspuru-Guzik -
Dmitry Zubarev -
Er, Suleyman -
Guerreschi, Gian -
Jacob Sanders -
Jarrod -
Joey Goodknight -
Salvatore Mandrà -
Sam Blau -
Stéphanie Valleau -
Tim Hirzel