CoreFreq is not compatible with BMQ

Created on 10 May 2019  路  20Comments  路  Source: cyring/CoreFreq

In kernels with the BMQ scheduler patch, struct task_struct does not have wake_cpu. This is problematic as the cpufreqk kernel module exposes it, and the userland utilities use it to index cTask (among others).

While I cannot make any good suggestions as to alternatives, struct task_struct.on_cpu is still available when BMQ is enabled (though it may be less correct, at least when BMQ isn't in use).

help wanted

All 20 comments

I have had success in replacing wake_cpu with on_cpu, though I'm not certain what effect it will have on the accuracy of measurements (if any).

It also appears that when BMQ is enabled, struct task_struct.se is not present (as with MUQSS). I updated the MUQSS config test macro to test CONFIG_SCHED_BMQ as well, which appears to work.

Thanks for sharing this configuration case.

For any low latency user reason, I've made the Tasks Monitoring optional in _CoreFreq_

Just start the daemon with the argument -goff

About building the driver in BMQ Kernel configuration , the BMQ macro directive could indeed leave the missing struct se fields with a zero value or with substituted values relevant for BMQ, and this, without introducing regressions in the rest of the software.

Edit: I will install BMQ and study the impacts

I'm using ArchLinux: is this the BMQ patch package linux-ck ?

Edit: because we have already worked on this. See issue #85 .

It's actually pf-kernel, so there may be some additional stuff in the way, but the CONFIG_SCHED_MUQSS test is also applicable to CONFIG_SCHED_BMQ (see my hack - though I think that there is a different, more applicable config flag that should be tested for this rather than CONFIG_SCHED_*).

Fyi, I have pulled the last linux-ck version and successfully build & run _CoreFreq_

$ make
cc  -Wall -pthread -c corefreqd.c \
        -D FEAT_DBG=1 -o corefreqd.o
cc  -Wall -c corefreqm.c -o corefreqm.o
cc  -Wall corefreqd.c corefreqm.c \
        -D FEAT_DBG=1 -o corefreqd -lpthread -lm -lrt
cc  -Wall -c corefreq-cli.c -o corefreq-cli.o
cc  -Wall -c corefreq-ui.c -o corefreq-ui.o
cc  -Wall -c corefreq-cli-rsc.c \
        -o corefreq-cli-rsc.o
cc  -Wall -c corefreq-cli-json.c \
        -o corefreq-cli-json.o
cc  -Wall -c corefreq-cli-extra.c \
        -o corefreq-cli-extra.o
cc  -Wall \
        corefreq-cli.c corefreq-ui.c corefreq-cli-rsc.c \
        corefreq-cli-json.c corefreq-cli-extra.c \
        -o corefreq-cli -lm -lrt
make -j1 -C /lib/modules/5.0.15-1-ck/build M=CoreFreq modules
make[1]: Entering directory '/usr/lib/modules/5.0.15-1-ck/build'
  CC [M]  CoreFreq/corefreqk.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC     CoreFreq/corefreqk.mod.o
  LD [M] CoreFreq/corefreqk.ko
make[1]: Leaving directory '/usr/lib/modules/5.0.15-1-ck/build'

2019-05-10-232032_804x644_scrot

OK, so only this change is enough to be compatible with your kernel config ?

...
#if defined(CONFIG_SCHED_MUQSS) || defined(CONFIG_SCHED_BMQ)
        SysGate->taskList[cnt].runtime  = tsk_seruntime(thread);
#else
...

Hello,
Is the latest version doing the job ?
Feel free to close the issue if ok.
Regards
Cyril

@cyring sorry for the delay. no, unfortunately that's just part of the required changes. the other issue is that of wake_cpu not being part of struct task_struct.

But how is defined your task_struct ?
The only condition of wake_cpu is CONFIG_SMP
Can you explain why this field is not part of the structure?

In linux-pf, there are a few other patches on top of BMQ that could relate to this, but wake_cpu's existence is determined by #ifndef CONFIG_SCHED_BMQ.

struct task_struct {
#ifdef CONFIG_THREAD_INFO_IN_TASK
    /*
     * For reasons of header soup (see current_thread_info()), this
     * must be the first element of task_struct.
     */
    struct thread_info      thread_info;
#endif
    /* -1 unrunnable, 0 runnable, >0 stopped: */
    volatile long           state;

    /*
     * This begins the randomizable portion of task_struct. Only
     * scheduling-critical items should be added above here.
     */
    randomized_struct_fields_start

    void                *stack;
    atomic_t            usage;
    /* Per task flags (PF_*), defined further below: */
    unsigned int            flags;
    unsigned int            ptrace;

#if defined(CONFIG_SMP) && !defined(CONFIG_SCHED_BMQ)
    struct llist_node       wake_entry;
#endif
#if defined(CONFIG_SMP) || defined(CONFIG_SCHED_BMQ)
    int             on_cpu;
#endif
#ifdef CONFIG_SMP
#ifdef CONFIG_THREAD_INFO_IN_TASK
    /* Current CPU: */
    unsigned int            cpu;
#endif
#ifndef CONFIG_SCHED_BMQ
    unsigned int            wakee_flips;
    unsigned long           wakee_flip_decay_ts;
    struct task_struct      *last_wakee;

    /*
     * recent_used_cpu is initially set as the last CPU used by a task
     * that wakes affine another task. Waker/wakee relationships can
     * push tasks around a CPU where each wakeup moves to the next one.
     * Tracking a recently used CPU allows a quick search for a recently
     * used CPU that may be idle.
     */
    int             recent_used_cpu;
    int             wake_cpu;
#endif /* !CONFIG_SCHED_BMQ */
#endif
    int             on_rq;

    int             prio;
    int             static_prio;
    int             normal_prio;
    unsigned int            rt_priority;

#ifdef CONFIG_SCHED_BMQ
    u64             last_ran;
    s64             time_slice;
    int             boost_prio;
    int             bmq_idx;
    struct list_head        bmq_node;
    /* sched_clock time spent running */
    u64             sched_time;
#else /* !CONFIG_SCHED_BMQ */
    const struct sched_class    *sched_class;
    struct sched_entity     se;
    struct sched_rt_entity      rt;
    struct sched_dl_entity      dl;
#endif
#ifdef CONFIG_CGROUP_SCHED
    struct task_group       *sched_task_group;
#endif

#ifdef CONFIG_PREEMPT_NOTIFIERS
    /* List of struct preempt_notifier: */
    struct hlist_head       preempt_notifiers;
#endif

#ifdef CONFIG_BLK_DEV_IO_TRACE
    unsigned int            btrace_seq;
#endif

    unsigned int            policy;
    int             nr_cpus_allowed;
    cpumask_t           cpus_allowed;

#ifdef CONFIG_PREEMPT_RCU
    int             rcu_read_lock_nesting;
    union rcu_special       rcu_read_unlock_special;
    struct list_head        rcu_node_entry;
    struct rcu_node         *rcu_blocked_node;
#endif /* #ifdef CONFIG_PREEMPT_RCU */

#ifdef CONFIG_TASKS_RCU
    unsigned long           rcu_tasks_nvcsw;
    u8              rcu_tasks_holdout;
    u8              rcu_tasks_idx;
    int             rcu_tasks_idle_cpu;
    struct list_head        rcu_tasks_holdout_list;
#endif /* #ifdef CONFIG_TASKS_RCU */

    struct sched_info       sched_info;

    struct list_head        tasks;
#ifdef CONFIG_SMP
    struct plist_node       pushable_tasks;
    struct rb_node          pushable_dl_tasks;
#endif

    struct mm_struct        *mm;
    struct mm_struct        *active_mm;

    /* Per-thread vma caching: */
    struct vmacache         vmacache;

#ifdef SPLIT_RSS_COUNTING
    struct task_rss_stat        rss_stat;
#endif
    int             exit_state;
    int             exit_code;
    int             exit_signal;
    /* The signal sent when the parent dies: */
    int             pdeath_signal;
    /* JOBCTL_*, siglock protected: */
    unsigned long           jobctl;

    /* Used for emulating ABI behavior of previous Linux versions: */
    unsigned int            personality;

    /* Scheduler bits, serialized by scheduler locks: */
    unsigned            sched_reset_on_fork:1;
    unsigned            sched_contributes_to_load:1;
    unsigned            sched_migrated:1;
    unsigned            sched_remote_wakeup:1;
#ifdef CONFIG_PSI
    unsigned            sched_psi_wake_requeue:1;
#endif

    /* Force alignment to the next boundary: */
    unsigned            :0;

    /* Unserialized, strictly 'current' */

    /* Bit to tell LSMs we're in execve(): */
    unsigned            in_execve:1;
    unsigned            in_iowait:1;
#ifndef TIF_RESTORE_SIGMASK
    unsigned            restore_sigmask:1;
#endif
#ifdef CONFIG_MEMCG
    unsigned            in_user_fault:1;
#endif
#ifdef CONFIG_COMPAT_BRK
    unsigned            brk_randomized:1;
#endif
#ifdef CONFIG_CGROUPS
    /* disallow userland-initiated cgroup migration */
    unsigned            no_cgroup_migration:1;
#endif
#ifdef CONFIG_BLK_CGROUP
    /* to be used once the psi infrastructure lands upstream. */
    unsigned            use_memdelay:1;
#endif

    unsigned long           atomic_flags; /* Flags requiring atomic access. */

    struct restart_block        restart_block;

    pid_t               pid;
    pid_t               tgid;

#ifdef CONFIG_STACKPROTECTOR
    /* Canary value for the -fstack-protector GCC feature: */
    unsigned long           stack_canary;
#endif
    /*
     * Pointers to the (original) parent process, youngest child, younger sibling,
     * older sibling, respectively.  (p->father can be replaced with
     * p->real_parent->pid)
     */

    /* Real parent process: */
    struct task_struct __rcu    *real_parent;

    /* Recipient of SIGCHLD, wait4() reports: */
    struct task_struct __rcu    *parent;

    /*
     * Children/sibling form the list of natural children:
     */
    struct list_head        children;
    struct list_head        sibling;
    struct task_struct      *group_leader;

    /*
     * 'ptraced' is the list of tasks this task is using ptrace() on.
     *
     * This includes both natural children and PTRACE_ATTACH targets.
     * 'ptrace_entry' is this task's link on the p->parent->ptraced list.
     */
    struct list_head        ptraced;
    struct list_head        ptrace_entry;

    /* PID/PID hash table linkage. */
    struct pid          *thread_pid;
    struct hlist_node       pid_links[PIDTYPE_MAX];
    struct list_head        thread_group;
    struct list_head        thread_node;

    struct completion       *vfork_done;

    /* CLONE_CHILD_SETTID: */
    int __user          *set_child_tid;

    /* CLONE_CHILD_CLEARTID: */
    int __user          *clear_child_tid;

    u64             utime;
    u64             stime;
#ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME
    u64             utimescaled;
    u64             stimescaled;
#endif
    u64             gtime;
    struct prev_cputime     prev_cputime;
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
    struct vtime            vtime;
#endif

#ifdef CONFIG_NO_HZ_FULL
    atomic_t            tick_dep_mask;
#endif
    /* Context switch counts: */
    unsigned long           nvcsw;
    unsigned long           nivcsw;

    /* Monotonic time in nsecs: */
    u64             start_time;

    /* Boot based time in nsecs: */
    u64             real_start_time;

    /* MM fault and swap info: this can arguably be seen as either mm-specific or thread-specific: */
    unsigned long           min_flt;
    unsigned long           maj_flt;

#ifdef CONFIG_POSIX_TIMERS
    struct task_cputime     cputime_expires;
    struct list_head        cpu_timers[3];
#endif

    /* Process credentials: */

    /* Tracer's credentials at attach: */
    const struct cred __rcu     *ptracer_cred;

    /* Objective and real subjective task credentials (COW): */
    const struct cred __rcu     *real_cred;

    /* Effective (overridable) subjective task credentials (COW): */
    const struct cred __rcu     *cred;

    /*
     * executable name, excluding path.
     *
     * - normally initialized setup_new_exec()
     * - access it with [gs]et_task_comm()
     * - lock it with task_lock()
     */
    char                comm[TASK_COMM_LEN];

    struct nameidata        *nameidata;

#ifdef CONFIG_SYSVIPC
    struct sysv_sem         sysvsem;
    struct sysv_shm         sysvshm;
#endif
#ifdef CONFIG_DETECT_HUNG_TASK
    unsigned long           last_switch_count;
    unsigned long           last_switch_time;
#endif
    /* Filesystem information: */
    struct fs_struct        *fs;

    /* Open file information: */
    struct files_struct     *files;

    /* Namespaces: */
    struct nsproxy          *nsproxy;

    /* Signal handlers: */
    struct signal_struct        *signal;
    struct sighand_struct       *sighand;
    sigset_t            blocked;
    sigset_t            real_blocked;
    /* Restored if set_restore_sigmask() was used: */
    sigset_t            saved_sigmask;
    struct sigpending       pending;
    unsigned long           sas_ss_sp;
    size_t              sas_ss_size;
    unsigned int            sas_ss_flags;

    struct callback_head        *task_works;

    struct audit_context        *audit_context;
#ifdef CONFIG_AUDITSYSCALL
    kuid_t              loginuid;
    unsigned int            sessionid;
#endif
    struct seccomp          seccomp;

    /* Thread group tracking: */
    u32             parent_exec_id;
    u32             self_exec_id;

    /* Protection against (de-)allocation: mm, files, fs, tty, keyrings, mems_allowed, mempolicy: */
    spinlock_t          alloc_lock;

    /* Protection of the PI data structures: */
    raw_spinlock_t          pi_lock;

    struct wake_q_node      wake_q;

#ifdef CONFIG_RT_MUTEXES
    /* PI waiters blocked on a rt_mutex held by this task: */
    struct rb_root_cached       pi_waiters;
    /* Updated under owner's pi_lock and rq lock */
    struct task_struct      *pi_top_task;
    /* Deadlock detection and priority inheritance handling: */
    struct rt_mutex_waiter      *pi_blocked_on;
#endif

#ifdef CONFIG_DEBUG_MUTEXES
    /* Mutex deadlock detection: */
    struct mutex_waiter     *blocked_on;
#endif

#ifdef CONFIG_TRACE_IRQFLAGS
    unsigned int            irq_events;
    unsigned long           hardirq_enable_ip;
    unsigned long           hardirq_disable_ip;
    unsigned int            hardirq_enable_event;
    unsigned int            hardirq_disable_event;
    int             hardirqs_enabled;
    int             hardirq_context;
    unsigned long           softirq_disable_ip;
    unsigned long           softirq_enable_ip;
    unsigned int            softirq_disable_event;
    unsigned int            softirq_enable_event;
    int             softirqs_enabled;
    int             softirq_context;
#endif

#ifdef CONFIG_LOCKDEP
# define MAX_LOCK_DEPTH         48UL
    u64             curr_chain_key;
    int             lockdep_depth;
    unsigned int            lockdep_recursion;
    struct held_lock        held_locks[MAX_LOCK_DEPTH];
#endif

#ifdef CONFIG_UBSAN
    unsigned int            in_ubsan;
#endif

    /* Journalling filesystem info: */
    void                *journal_info;

    /* Stacked block device info: */
    struct bio_list         *bio_list;

#ifdef CONFIG_BLOCK
    /* Stack plugging: */
    struct blk_plug         *plug;
#endif

    /* VM state: */
    struct reclaim_state        *reclaim_state;

    struct backing_dev_info     *backing_dev_info;

    struct io_context       *io_context;

    /* Ptrace state: */
    unsigned long           ptrace_message;
    kernel_siginfo_t        *last_siginfo;

    struct task_io_accounting   ioac;
#ifdef CONFIG_PSI
    /* Pressure stall state */
    unsigned int            psi_flags;
#endif
#ifdef CONFIG_TASK_XACCT
    /* Accumulated RSS usage: */
    u64             acct_rss_mem1;
    /* Accumulated virtual memory usage: */
    u64             acct_vm_mem1;
    /* stime + utime since last update: */
    u64             acct_timexpd;
#endif
#ifdef CONFIG_CPUSETS
    /* Protected by ->alloc_lock: */
    nodemask_t          mems_allowed;
    /* Seqence number to catch updates: */
    seqcount_t          mems_allowed_seq;
    int             cpuset_mem_spread_rotor;
    int             cpuset_slab_spread_rotor;
#endif
#ifdef CONFIG_CGROUPS
    /* Control Group info protected by css_set_lock: */
    struct css_set __rcu        *cgroups;
    /* cg_list protected by css_set_lock and tsk->alloc_lock: */
    struct list_head        cg_list;
#endif
#ifdef CONFIG_X86_CPU_RESCTRL
    u32             closid;
    u32             rmid;
#endif
#ifdef CONFIG_FUTEX
    struct robust_list_head __user  *robust_list;
#ifdef CONFIG_COMPAT
    struct compat_robust_list_head __user *compat_robust_list;
#endif
    struct list_head        pi_state_list;
    struct futex_pi_state       *pi_state_cache;
#endif
#ifdef CONFIG_PERF_EVENTS
    struct perf_event_context   *perf_event_ctxp[perf_nr_task_contexts];
    struct mutex            perf_event_mutex;
    struct list_head        perf_event_list;
#endif
#ifdef CONFIG_DEBUG_PREEMPT
    unsigned long           preempt_disable_ip;
#endif
#ifdef CONFIG_NUMA
    /* Protected by alloc_lock: */
    struct mempolicy        *mempolicy;
    short               il_prev;
    short               pref_node_fork;
#endif
#ifdef CONFIG_NUMA_BALANCING
    int             numa_scan_seq;
    unsigned int            numa_scan_period;
    unsigned int            numa_scan_period_max;
    int             numa_preferred_nid;
    unsigned long           numa_migrate_retry;
    /* Migration stamp: */
    u64             node_stamp;
    u64             last_task_numa_placement;
    u64             last_sum_exec_runtime;
    struct callback_head        numa_work;

    struct numa_group       *numa_group;

    /*
     * numa_faults is an array split into four regions:
     * faults_memory, faults_cpu, faults_memory_buffer, faults_cpu_buffer
     * in this precise order.
     *
     * faults_memory: Exponential decaying average of faults on a per-node
     * basis. Scheduling placement decisions are made based on these
     * counts. The values remain static for the duration of a PTE scan.
     * faults_cpu: Track the nodes the process was running on when a NUMA
     * hinting fault was incurred.
     * faults_memory_buffer and faults_cpu_buffer: Record faults per node
     * during the current scan window. When the scan completes, the counts
     * in faults_memory and faults_cpu decay and these values are copied.
     */
    unsigned long           *numa_faults;
    unsigned long           total_numa_faults;

    /*
     * numa_faults_locality tracks if faults recorded during the last
     * scan window were remote/local or failed to migrate. The task scan
     * period is adapted based on the locality of the faults with different
     * weights depending on whether they were shared or private faults
     */
    unsigned long           numa_faults_locality[3];

    unsigned long           numa_pages_migrated;
#endif /* CONFIG_NUMA_BALANCING */

#ifdef CONFIG_RSEQ
    struct rseq __user *rseq;
    u32 rseq_len;
    u32 rseq_sig;
    /*
     * RmW on rseq_event_mask must be performed atomically
     * with respect to preemption.
     */
    unsigned long rseq_event_mask;
#endif

    struct tlbflush_unmap_batch tlb_ubc;

    struct rcu_head         rcu;

    /* Cache last used pipe for splice(): */
    struct pipe_inode_info      *splice_pipe;

    struct page_frag        task_frag;

#ifdef CONFIG_TASK_DELAY_ACCT
    struct task_delay_info      *delays;
#endif

#ifdef CONFIG_FAULT_INJECTION
    int             make_it_fail;
    unsigned int            fail_nth;
#endif
    /*
     * When (nr_dirtied >= nr_dirtied_pause), it's time to call
     * balance_dirty_pages() for a dirty throttling pause:
     */
    int             nr_dirtied;
    int             nr_dirtied_pause;
    /* Start of a write-and-pause period: */
    unsigned long           dirty_paused_when;

#ifdef CONFIG_LATENCYTOP
    int             latency_record_count;
    struct latency_record       latency_record[LT_SAVECOUNT];
#endif
    /*
     * Time slack values; these are used to round up poll() and
     * select() etc timeout values. These are in nanoseconds.
     */
    u64             timer_slack_ns;
    u64             default_timer_slack_ns;

#ifdef CONFIG_KASAN
    unsigned int            kasan_depth;
#endif

#ifdef CONFIG_FUNCTION_GRAPH_TRACER
    /* Index of current stored address in ret_stack: */
    int             curr_ret_stack;
    int             curr_ret_depth;

    /* Stack of return addresses for return function tracing: */
    struct ftrace_ret_stack     *ret_stack;

    /* Timestamp for last schedule: */
    unsigned long long      ftrace_timestamp;

    /*
     * Number of functions that haven't been traced
     * because of depth overrun:
     */
    atomic_t            trace_overrun;

    /* Pause tracing: */
    atomic_t            tracing_graph_pause;
#endif

#ifdef CONFIG_TRACING
    /* State flags for use by tracers: */
    unsigned long           trace;

    /* Bitmask and counter of trace recursion: */
    unsigned long           trace_recursion;
#endif /* CONFIG_TRACING */

#ifdef CONFIG_KCOV
    /* Coverage collection mode enabled for this task (0 if disabled): */
    unsigned int            kcov_mode;

    /* Size of the kcov_area: */
    unsigned int            kcov_size;

    /* Buffer for coverage collection: */
    void                *kcov_area;

    /* KCOV descriptor wired with this task or NULL: */
    struct kcov         *kcov;
#endif

#ifdef CONFIG_MEMCG
    struct mem_cgroup       *memcg_in_oom;
    gfp_t               memcg_oom_gfp_mask;
    int             memcg_oom_order;

    /* Number of pages to reclaim on returning to userland: */
    unsigned int            memcg_nr_pages_over_high;

    /* Used by memcontrol for targeted memcg charge: */
    struct mem_cgroup       *active_memcg;
#endif

#ifdef CONFIG_BLK_CGROUP
    struct request_queue        *throttle_queue;
#endif

#ifdef CONFIG_UPROBES
    struct uprobe_task      *utask;
#endif
#if defined(CONFIG_BCACHE) || defined(CONFIG_BCACHE_MODULE)
    unsigned int            sequential_io;
    unsigned int            sequential_io_avg;
#endif
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
    unsigned long           task_state_change;
#endif
    int             pagefault_disabled;
#ifdef CONFIG_MMU
    struct task_struct      *oom_reaper_list;
#endif
#ifdef CONFIG_VMAP_STACK
    struct vm_struct        *stack_vm_area;
#endif
#ifdef CONFIG_THREAD_INFO_IN_TASK
    /* A live task holds one reference: */
    atomic_t            stack_refcount;
#endif
#ifdef CONFIG_LIVEPATCH
    int patch_state;
#endif
#ifdef CONFIG_SECURITY
    /* Used by LSM modules for access restriction: */
    void                *security;
#endif

#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
    unsigned long           lowest_stack;
    unsigned long           prev_lowest_stack;
#endif

    /*
     * New fields for task_struct should be added above here, so that
     * they are included in the randomized portion of task_struct.
     */
    randomized_struct_fields_end

    /* CPU-specific state of this task: */
    struct thread_struct        thread;

    /*
     * WARNING: on x86, 'thread_struct' contains a variable-sized
     * structure.  It *MUST* be at the end of 'task_struct'.
     *
     * Do not put anything below here!
     */
};

To clarify, I did modify CoreFreq to use on_cpu, but as stated previously I'm unsure of the effect that will have vs wake_cpu primarily due to a (apparent) lack of good documentation for the members of task_struct.

How does the top command run w/ this patch ?
If it correctly shows the last cpu the task was on, then it means that somewhere, between the task structure and top, some code is handling the case.

The same, If on_cpu is coherent with the results of any scheduling tracer, such as /proc/sched_debug

I'll have a look at the top source code. Unfortunately, I don't have sched_debug in my procfs but that may be down to my disabling it (it probably is). I'll have a look at the code for that as well.

OK. Just briefly looking at it, it appears to be that on_cpu is usually 0 (although i have seen tasks appear on other CPUs as well).

Here is running zpaq on a large file with seven threads. You can see that some tasks are shown on CPU1 but almost all are on CPU0, you can see that the threads are distributed (or are most active) across CPU0-3

zpaq -t7

Here is zpaq running with a single thread and 100% affinity for CPU7. You can see that CPU7 is busy but that the tasks are still reported on the incorrect CPU.

taskset -c 7 zpaq -t1

Based on this, on_cpu is _not_ an accurate value. I'll have a read through BMQ scheduler code and see if I can glean any understanding of how (and where) it tracks CPU migration.

Also note - BMQ is a continuation of (though still different from) PDS, if you ever had workarounds for PDS.

After a brief inspection, it appears that BMQ tracks the CPU for a given task in struct_task.cpu, unless CONFIG_THREAD_INFO_IN_TASK in which case it is task_thread_info(struct_task)->cpu (which is (((struct thread_info *)(task)->stack))->cpu on every arch except ia64).

My understanding of this is per kernel/sched/bmq.c in function __set_task_cpu

static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)
{
#ifdef CONFIG_SMP
    /*
     * After ->cpu is set up to a new value, task_access_lock(p, ...) can be
     * successfully executed on another CPU. We must ensure that updates of
     * per-task data have been completed by this moment.
     */
    smp_wmb();

#ifdef CONFIG_THREAD_INFO_IN_TASK
    p->cpu = cpu;
#else
    task_thread_info(p)->cpu = cpu;
#endif
#endif
}

Note the feature test for CONFIG_THREAD_INFO_IN_TASK at the end - this is exposed in include/linux/sched.h as unsigned int task_cpu

/*
 * Wrappers for p->thread_info->cpu access. No-op on UP.
 */
#ifdef CONFIG_SMP

static inline unsigned int task_cpu(const struct task_struct *p)
{
#ifdef CONFIG_THREAD_INFO_IN_TASK
    return READ_ONCE(p->cpu);
#else
    return READ_ONCE(task_thread_info(p)->cpu);
#endif
}

extern void set_task_cpu(struct task_struct *p, unsigned int cpu);

#else

Applying the relevant changes to the kernel module yields promising results -

corefreqk after using ->cpu

Congratulations.

Do you have the change available in your _CoreFreq_ fork ?

I'm reading above:

/*
 * Wrappers for p->thread_info->cpu access. No-op on UP.
 */

Does that mean the CPU will be undefined if Kernel is booted with a single Core ?

PDS ?

Also notice the limits:

  • Max number of tasks (stored in shared memory)

https://github.com/cyring/CoreFreq/blob/124d9e54ab2b5b0385855745d416d93b57346cfa/coretypes.h#L1435

You can gently increase the order if needed. Issue #45

@cyring I may have pushed it to my fork. Though I want to do some cleanup before submitting a PR.

Does that mean the CPU will be undefined if Kernel is booted with single Core ?

I believe it refers to what happens if !defined(CONFIG_SMP).

Hello,
Can we backport the adapted code ?

@cyring Hey, sorry about the silence. I've been very busy. Yes - we can backport it. I'll make a point of preparing a PR tonight.

Apologies.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

EUA picture EUA  路  8Comments

vincenzogianfelice picture vincenzogianfelice  路  17Comments

ich777 picture ich777  路  10Comments

thor2002ro picture thor2002ro  路  22Comments

inglor picture inglor  路  16Comments